Webhooks
Get finished batch jobs, crawls and new monitor results as signed POSTs: payload, headers, HMAC-SHA256 checks in Node.js, Python, Go and PHP, and retries.
When a batch job or a crawl finishes, or a monitor finds new results, SerpKite can POST them to a URL you control instead of waiting for you to poll. Deliveries are signed with your account’s webhook secret so you can prove they came from SerpKite.
Set it up
-
Create a webhook secret
Every delivery is signed. Your account gets a secret automatically with its first webhook; to see it, open Settings → Webhooks in the dashboard and click Rotate secret (Generate secret if none exists yet), which shows the new secret once. The secret starts with
whsec_; store it as an environment variable such asSERPKITE_WEBHOOK_SECRET. -
Choose where deliveries go
Either set a default Webhook URL in the same settings screen (used by every batch job), or pass
webhook_urlon thePOST /v1/batchescall, which takes precedence for every job in that batch:curl https://api.serpkite.com/v1/batches \ -H "Authorization: Bearer $SERPKITE_API_KEY" \ -H "Content-Type: application/json" \ -d '{"endpoint":"search","requests":[{"q":"espresso grinder","num":100}],"webhook_url":"https://example.com/hooks/serpkite"}'POST /v1/crawltakeswebhook_urlthe same way (falling back to the account URL), and every monitor has its own optionalwebhook_url. Realtime endpoints such as/v1/searchreturn the result in the response and don’t acceptwebhook_url. -
Verify and acknowledge
Your endpoint checks the signature, stores the result, and answers with any
2xxstatus within 10 seconds. Do slow work after responding.
The delivery
SerpKite sends one POST per finished job, whether it succeeded or failed.
| Header | Value |
|---|---|
Content-Type |
application/json |
User-Agent |
SerpKite-Webhooks/1.0 |
X-SerpKite-Event |
batch.completed |
X-SerpKite-Delivery |
The job ID. The same on every retry, so use it to deduplicate. |
X-SerpKite-Timestamp |
Unix time (seconds) when this attempt was signed. |
X-SerpKite-Signature |
v1= followed by the hex HMAC-SHA256 signature. |
The body is the job, with the result inline. Fields: event, id, status (done or failed), endpoint, created_at, completed_at, credits_used, and either result (the same body the realtime endpoint returns) or error.
{
"event": "batch.completed",
"id": "0192f7a4-6c1e-7b3a-9d2f-5e8a1c4b7d90",
"status": "done",
"endpoint": "/v1/search",
"created_at": "2026-09-29T08:00:00Z",
"completed_at": "2026-09-29T08:03:12Z",
"credits_used": 2.5,
"result": {
"request": { "endpoint": "search", "engine": "google", "q": "espresso grinder", "num": 50 },
"results": ["…"],
"related_searches": [],
"meta": { "request_id": "req_01J8ZK4M6Q2V7", "credits_used": 2.5, "cached": false }
}
}
When status is failed, there is no result; instead error holds code and message, and the job’s credits were refunded. The result is also available from GET /v1/batches/{id} for 24 hours, so a lost delivery is never lost data.
Other events
Crawls, research tasks and monitors use the same headers and signature; only X-SerpKite-Event and the body differ.
X-SerpKite-Event |
Sent when | X-SerpKite-Delivery |
Body |
|---|---|---|---|
batch.completed |
A batch job finished | The job ID | The job, as above |
crawl.completed |
A crawl completed, failed or was canceled | The crawl ID | event, id, kind, status, created_at, completed_at, credits_used, poll_url, and result (the pages; omitted with result_omitted: true above 4 MB, fetch it from poll_url) or error |
research.completed |
A research task completed, failed or was canceled | The task ID | Same shape as crawl.completed, with kind: "research" and result holding the report, citations, sources, queries, stats and, with output_schema, output |
monitor.results |
A monitor run found results it had not seen before; an AI assistant monitor sends every run | The run ID | event, monitor_id, run_id, name, endpoint, q (or url for page monitors, prompt for AI assistant monitors), metadata, first_run, run_at, new_results, credits_used. For an AI assistant monitor, new_results holds the run’s answer: text, markdown, model, sources, new_sources, search_queries, products and, with track, mentions |
Crawl and research deliveries are retried on the schedule below. A monitor tries its webhook up to 3 times per run; results it could not deliver stay new and are sent again with the next run, and every run’s new results are also in its run history for 24 hours.
Verifying the signature
The signature is an HMAC-SHA256 over the timestamp, a dot, and the raw request body, keyed with your webhook secret:
X-SerpKite-Signature: v1=hex( HMAC_SHA256( secret, timestamp + "." + raw_body ) )
To verify:
- Read the raw body bytes before any JSON parsing. Re-serialised JSON will not match.
- Compute the HMAC with your secret and compare it to the header using a constant-time comparison.
- Reject timestamps older than a few minutes (5 minutes is a good default) to block replays.
- Deduplicate on
X-SerpKite-Delivery, since a retry can arrive after you already processed the job.
With an SDK
The official SDKs verify the signature and the timestamp (5-minute tolerance) for you. Pass the raw body:
import { verifyWebhook } from "serpkite";
const ok = await verifyWebhook(process.env.SERPKITE_WEBHOOK_SECRET!, rawBody, req.headers);
from serpkite import verify_webhook
ok = verify_webhook(os.environ["SERPKITE_WEBHOOK_SECRET"], request.get_data(), request.headers)
ok := serpkite.VerifyWebhook(os.Getenv("SERPKITE_WEBHOOK_SECRET"), body, r.Header, time.Now())
Node.js
import crypto from "node:crypto";
import express from "express";
const app = express();
const SECRET = process.env.SERPKITE_WEBHOOK_SECRET;
app.post("/hooks/serpkite", express.raw({ type: "application/json" }), (req, res) => {
const ts = req.get("X-SerpKite-Timestamp") ?? "";
const sig = req.get("X-SerpKite-Signature") ?? "";
const expected =
"v1=" + crypto.createHmac("sha256", SECRET).update(`${ts}.`).update(req.body).digest("hex");
const fresh = Math.abs(Date.now() / 1000 - Number(ts)) < 300;
const valid =
sig.length === expected.length && crypto.timingSafeEqual(Buffer.from(sig), Buffer.from(expected));
if (!fresh || !valid) return res.status(401).end();
const job = JSON.parse(req.body.toString("utf8"));
res.status(204).end(); // acknowledge first, then process
handleJob(req.get("X-SerpKite-Delivery"), job);
});
Python
import hashlib, hmac, os, time
from flask import Flask, request, abort
app = Flask(__name__)
SECRET = os.environ["SERPKITE_WEBHOOK_SECRET"].encode()
@app.post("/hooks/serpkite")
def serpkite_hook():
ts = request.headers.get("X-SerpKite-Timestamp", "")
sig = request.headers.get("X-SerpKite-Signature", "")
raw = request.get_data() # raw bytes, not request.json
expected = "v1=" + hmac.new(SECRET, ts.encode() + b"." + raw, hashlib.sha256).hexdigest()
if not ts.isdigit() or abs(time.time() - int(ts)) > 300:
abort(401)
if not hmac.compare_digest(sig, expected):
abort(401)
job = request.get_json()
enqueue(request.headers["X-SerpKite-Delivery"], job) # your own queue
return "", 204
Go
package hooks
import (
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"io"
"math"
"net/http"
"os"
"strconv"
"time"
)
var secret = []byte(os.Getenv("SERPKITE_WEBHOOK_SECRET"))
func SerpKite(w http.ResponseWriter, r *http.Request) {
body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 32<<20))
if err != nil {
http.Error(w, "bad body", http.StatusBadRequest)
return
}
ts := r.Header.Get("X-SerpKite-Timestamp")
sec, err := strconv.ParseInt(ts, 10, 64)
if err != nil || math.Abs(float64(time.Now().Unix()-sec)) > 300 {
http.Error(w, "stale", http.StatusUnauthorized)
return
}
mac := hmac.New(sha256.New, secret)
mac.Write([]byte(ts + "."))
mac.Write(body)
expected := "v1=" + hex.EncodeToString(mac.Sum(nil))
if !hmac.Equal([]byte(expected), []byte(r.Header.Get("X-SerpKite-Signature"))) {
http.Error(w, "bad signature", http.StatusUnauthorized)
return
}
w.WriteHeader(http.StatusNoContent)
go process(r.Header.Get("X-SerpKite-Delivery"), body)
}
PHP
<?php
$secret = getenv('SERPKITE_WEBHOOK_SECRET');
$raw = file_get_contents('php://input');
$ts = $_SERVER['HTTP_X_SERPKITE_TIMESTAMP'] ?? '';
$sig = $_SERVER['HTTP_X_SERPKITE_SIGNATURE'] ?? '';
$expected = 'v1=' . hash_hmac('sha256', $ts . '.' . $raw, $secret);
if (!ctype_digit($ts) || abs(time() - (int) $ts) > 300 || !hash_equals($expected, $sig)) {
http_response_code(401);
exit;
}
$job = json_decode($raw, true);
http_response_code(204);
// store $job keyed by $_SERVER['HTTP_X_SERPKITE_DELIVERY']
Retries
A delivery succeeds when your endpoint answers with any 2xx status within 10 seconds. Anything else (a timeout, a connection error, a 3xx, 4xx or 5xx) counts as a failure and is retried with exponential backoff: about 1 minute, then 2, 4, 8 and so on, up to roughly an hour apart, for up to 8 attempts in total.
The job’s webhook_status tracks the outcome and is visible on GET /v1/batches/{id} and in the dashboard:
webhook_status |
Meaning |
|---|---|
pending |
Not delivered yet, or waiting for the next retry. |
delivered |
Your endpoint returned 2xx. |
failed |
All attempts failed. Fetch the result by polling within 24 hours. |
Each retry is signed again with a fresh timestamp, so always verify against the headers of the request you received.
Rotating the secret
Generate a new secret under Settings → Webhooks at any time. The new secret takes effect for the next delivery attempt and the old one stops working, so deploy the new value to your receiver first (accepting either secret for a short overlap), then rotate.
Troubleshooting
- Signature never matches: you are hashing parsed-and-reserialised JSON, or a framework decompressed or re-encoded the body. Hash the exact bytes you received.
- Duplicate processing: retries reuse
X-SerpKite-Delivery; store processed IDs and skip repeats. - Deliveries time out: respond before doing the work. Results for
num: 100orinclude_contentcan be large, so allow bodies of several megabytes. - Local development: expose your machine with a tunnel (for example
cloudflared tunnelorngrok) and use that URL aswebhook_url.
Related: Batch requests, Batches endpoint, Spend controls.
Last updated: 2026-10-28