Crawl

Crawl reads a whole site, or one section of it, in the background. It starts at url, follows links on the same host (optionally subdomains) up to max_depth hops, and unless sitemap is skip also reads the URLs from the site's sitemaps. Each page comes back as Markdown (or text) with its metadata, through the same fetch as /v1/webpage, PDFs included.

It is polite by default: every host's robots.txt is honoured (RFC 9309 rules and Crawl-delay; a robots.txt that answers 5xx stops the crawl at the start page), URLs are canonicalised (fragments and tracking parameters dropped, www., trailing slashes and redirects folded) so each page is read once, and at most 5 pages are fetched at a time. Breadth first, or best first with query: pages whose URL and link text match it are read first.

POST /v1/crawl returns 202 with a task. Poll GET /v1/crawl/{id} (progress while it runs, result when it ends; kept 24 hours) or receive the signed crawl.completed webhook. DELETE /v1/crawl/{id} cancels it: a queued crawl is refunded at once, a running one stops at its next step and is charged for the pages read.

Request body

Example request

Authenticate with Authorization: Bearer $SERPKITE_API_KEY (GET requests may pass ?api_key= instead). The official SDKs read SERPKITE_API_KEY for you. See Authentication.

curl https://api.serpkite.com/v1/crawl \
  -H "Authorization: Bearer $SERPKITE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://docs.python.org/3/library/","limit":25,"max_depth":2,"include_paths":["^/3/library/asyncio"]}'

Response fields

Every billed response also carries credit and latency headers (X-Credits-Used, X-Credits-Remaining, X-Request-Id…).

Example response

{
  "id": "01926a3e-7b2c-7c4e-9f1a-2b3c4d5e6f70",
  "kind": "crawl",
  "status": "completed",
  "created_at": "2026-10-03T12:00:00Z",
  "started_at": "2026-10-03T12:00:01Z",
  "completed_at": "2026-10-03T12:00:41Z",
  "credits_reserved": 25,
  "credits_used": 18,
  "webhook_status": null,
  "progress": {
    "pages_done": 18,
    "pages_failed": 0,
    "pages_queued": 0,
    "pages_discovered": 18,
    "limit": 25
  },
  "result": {
    "url": "https://docs.python.org/3/library/",
    "pages": [
      {
        "url": "https://docs.python.org/3/library/asyncio.html",
        "depth": 1,
        "title": "asyncio — Asynchronous I/O",
        "markdown": "# asyncio — Asynchronous I/O\n\nasyncio is a library to write concurrent code…"
      }
    ],
    "failed": [],
    "stats": {
      "pages": 18,
      "failed": 0,
      "seconds": 40,
      "discovered": 18,
      "queued": 0,
      "duplicates": 1,
      "sitemap_urls": 0,
      "robots": "found",
      "robots_disallowed": 0,
      "robots_blocked": 0,
      "crawl_delay_ms": 0,
      "stopped": "done"
    }
  },
  "error": null
}

Errors

Errors use one shape: {"error":{"code","message","request_id"}}. Errors are never billed. Full list in Errors.

Notes

• A crawl runs for at most 30 minutes and returns up to 16 MB of pages; result.stats.stopped says why it ended (done, limit, time_limit, size_limit, too_many_failures, canceled).
• At most 20 crawls can be queued or running per account (429 beyond that). A crawl interrupted by a deploy goes back to the queue and restarts from the beginning.
• The remote MCP server has crawl and crawl_result tools.

Related

• Site ingestion guide
• Map
• Webhooks
• OpenAPI spec