How is billing calculated?

1 credit per page read (0.5 from cache). The limit is reserved when you start the crawl; when it ends you're charged for the pages actually read and the rest is refunded. A crawl that reads nothing is free. From $0.60 per 1,000 pages.

Does it respect robots.txt?

Yes. Every host's robots.txt is honoured, including Crawl-delay, and a robots.txt that answers 5xx stops the crawl at the start page. Only public pages are read; nothing behind a login.

How big can a crawl be?

Up to 1,000 pages, 10 link hops and 16 MB of results, for at most 30 minutes. Up to 20 crawls can run at once per account.

Can it render JavaScript pages?

Pages that need JavaScript, or a browser check a standard browser passes, are rendered in a browser when there's time. Pages behind a CAPTCHA are reported as failed and not charged.

How long are results kept?

24 hours after the crawl ends. Download them, or have them pushed to your webhook.

POST /v1/crawl

Website Crawler API

Give it a start URL and a page limit. SerpKite follows links and sitemaps in the background, reads each page as Markdown with its metadata, and calls your webhook when the crawl is done.

Get a free API key Read the reference

1 credit per page read (0.5 from cache); the limit is reserved up front and the unused part refunded 2,500 free credits, no card Failed calls are free

Overview

What the Website Crawler API does

The Website Crawler API reads a whole site, or one section of it, asynchronously. It starts at url, follows links on the same host up to max_depth hops (at most 10), reads the URLs in the site's sitemaps, and returns each page as Markdown or text with its title, publish date and metadata. PDFs are read too.

It is polite by default: robots.txt rules and Crawl-delay are honoured, URLs are canonicalised so each page is read once, and at most 5 pages are fetched at a time. With query, pages whose URL and link text match it are read first, so a 50-page limit spends itself on the pages you care about. include_paths and exclude_paths (regexes) keep the crawl inside a section.

POST /v1/crawl answers 202 with a task. Poll GET /v1/crawl/{id} or receive a signed crawl.completed webhook. limit credits are reserved up front and the difference is refunded when the crawl ends.

What you get

id string: The crawl's ID. poll_url is https://api.serpkite.com/v1/crawl/{id}.
status string: queued, running, completed, failed or canceled.
credits_reserved / credits_used number: The reservation (limit credits) and what the pages read cost; the difference is refunded when the crawl ends.
progress object: While running: pages_done, pages_failed, pages_queued, pages_discovered, limit.
result object: When it ends: url, pages[] (url, depth, title, markdown or text, published_at, metadata, links), failed[] (url, error) and stats (pages, failed, seconds, discovered, queued, duplicates, sitemap_urls, robots, robots_disallowed, robots_blocked, crawl_delay_ms, stopped).
error object | null: Set when the crawl failed (no_pages: nothing could be read, refunded) or was canceled.

Request

POST /v1/crawl

cURL

Python

TypeScript

fetch

PHP

curl https://api.serpkite.com/v1/crawl \
  -H "Authorization: Bearer $SERPKITE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://docs.python.org/3/library/","limit":25,"max_depth":2,"include_paths":["^/3/library/asyncio"]}'

Response (illustrative, placeholder domains)

GET /v1/crawl/{id} · 200 OK

{
  "id": "01926a3e-7b2c-7c4e-9f1a-2b3c4d5e6f70",
  "kind": "crawl",
  "status": "completed",
  "created_at": "2026-10-03T12:00:00Z",
  "started_at": "2026-10-03T12:00:01Z",
  "completed_at": "2026-10-03T12:00:41Z",
  "credits_reserved": 25,
  "credits_used": 18,
  "webhook_status": null,
  "progress": {
    "pages_done": 18,
    "pages_failed": 0,
    "pages_queued": 0,
    "pages_discovered": 18,
    "limit": 25
  },
  "result": {
    "url": "https://docs.python.org/3/library/",
    "pages": [
      {
        "url": "https://docs.python.org/3/library/asyncio.html",
        "depth": 1,
        "title": "asyncio — Asynchronous I/O",
        "markdown": "# asyncio — Asynchronous I/O\n\nasyncio is a library to write concurrent code…"
      }
    ],
    "failed": [],
    "stats": {
      "pages": 18,
      "failed": 0,
      "seconds": 40,
      "discovered": 18,
      "queued": 0,
      "duplicates": 1,
      "sitemap_urls": 0,
      "robots": "found",
      "robots_disallowed": 0,
      "robots_blocked": 0,
      "crawl_delay_ms": 0,
      "stopped": "done"
    }
  },
  "error": null
}

Every response also carries X-Credits-Used, X-Credits-Remaining, X-Cache and X-Tokens-Estimate headers. Full endpoint reference

Reference

Parameters

The fields /v1/crawl accepts in its JSON body.

Website Crawler API parameters
Name	Type	Default	Description
`url` required	string	–	Start page. Always read; path filters and robots.txt apply to the pages found from it.
`limit`	integer	25	Maximum pages read, 1–1,000. This many credits are reserved up front.
`max_depth`	integer	2	Link hops from url, 0–10 (sitemap pages count as 1 hop; 0 reads only url).
`include_paths`	string[]	–	Up to 20 regular expressions matched against the URL path; follow only paths matching one, e.g. ^/docs/.
`exclude_paths`	string[]	–	Up to 20 regular expressions matched against the URL path; never follow paths matching one.
`include_subdomains`	boolean	false	Also crawl subdomains of the start page's host.
`sitemap`	string	include	include also seeds the crawl with the site's sitemap URLs (after the start page's links), only reads the start page and sitemap URLs without following links, skip follows links only. Values: include, only, skip.
`query`	string	–	Read the pages most relevant to these words first (best-first crawl, up to 512 characters).
`ignore_query_parameters`	boolean	false	Treat URLs that differ only in their query string as one page.
`format`	string	markdown	What each page carries. Values: markdown, text.
`include_links`	boolean	false	Return each page's outbound links.
`max_tokens`	integer	–	Trim each page to about this many tokens (100–100,000).
`max_age`	integer	–	Accept cached pages up to this many seconds old, for 0.5 credit each.
`webhook_url`	string	–	Receives the signed crawl.completed event. Defaults to the account webhook URL.

Full reference, error codes and headers are in the /v1/crawl reference.

Use cases

What people build with the Website Crawler API

Knowledge bases

Load a product's docs or help centre into a vector store as clean Markdown chunks.

Agent research

Let an agent read a whole site section through the MCP crawl and crawl_result tools.

Content audits

Pull every page's title, dates and links to audit a site's content or structure.

Scheduled snapshots

Re-crawl a section nightly and diff the Markdown to see what changed.

Pricing