Map, extract, crawl and monitor the web with SerpKite

Search finds the sources. An agent still needs to open them, pick useful passages, read the rest of a documentation site, and come back when something changes. SerpKite now handles those steps with four new APIs: map, extract, crawl and monitors. They use the same API key and credit balance as search, with support in the TypeScript, Python and Go SDKs.

This release also adds PDF reading, links and images from pages, domain and date filters, and query-ranked highlights. Here is how the pieces fit together.

Choose the next step

API	Use it when	Credits
Extract /v1/extract	You already have the URLs and want their content or relevant passages.	1 credit per URL that comes back (0.5 from cache); failed URLs are free
Map /v1/map	You need a site's URLs before choosing what to read.	1 credit per call that found URLs (free when nothing is found)
Crawl /v1/crawl	You want to read a whole site or a section in the background.	1 credit per page read (0.5 from cache); the limit is reserved up front and the unused part refunded
Monitors /v1/monitors	You want recurring searches or page checks and a history of changes.	Billed per run, priced like the search or page fetch it makes (1 per 10 results, num=100 is 7); empty and failed runs are free

For a question about the wider web, start with search. For a question about a known documentation site, start with map. For a fixed list of sources, go straight to extract.

Discover URLs, then read the useful pages

POST /v1/map collects URLs from a site’s sitemaps and the links on its start page. It cleans and de-duplicates them, and can rank them against a search phrase. include_paths and exclude_paths are regular expressions on the URL path, so you can keep a documentation section and skip unrelated pages.

Map returns URLs; use POST /v1/extract to read the ones you choose. Extract accepts up to 20 URLs per call and returns Markdown, text or HTML, along with a separate failed list. One unreachable page does not discard the pages that succeeded.

Here is a small Python workflow that finds asyncio documentation and reads up to five pages:

pip install --upgrade serpkite
export SERPKITE_API_KEY=skt_live_...

from serpkite import SerpKite

with SerpKite() as sk:
    discovered = sk.map(
        "https://docs.python.org/3/",
        search="asyncio task cancellation",
        include_paths=[r"^/3/library/asyncio"],
        limit=5,
    )
    urls = [item.url for item in discovered.results]

    if urls:
        extracted = sk.extract(
            urls,
            query="How does asyncio task cancellation work?",
            highlights=3,
            include_links=True,
        )
        for page in extracted.results:
            print(page.url, page.highlights)
        for failure in extracted.failed:
            print("Could not read:", failure.url, failure.error.code)

The query and highlights options add passages ranked by BM25, a lexical ranking algorithm. Each highlight carries its text, score and heading. Your model can use the selected passages and their source URLs as evidence, or read the full Markdown when it needs more context. SerpKite retrieves and ranks the content; your application generates the answer.

Extract defaults to a 50-second deadline, configurable up to 90 seconds. Pages still loading at the deadline are reported as failures and cost nothing. See the extract reference for response fields and limits.

Read PDFs, links and images

Both webpage and extract can read public PDFs as well as HTML. PDF text is returned page by page, with page markers in Markdown and a page count in metadata. This reads the PDF’s text layer; a scanned document without a text layer returns empty and is not billed.

Set include_links: true to collect outgoing links, or include_images: true to collect image URLs. On webpage, these lists are also included in compact and Markdown output. The country option on webpage and extract chooses the exit country for pages whose content varies by region.

Crawl a documentation section in the background

When you need more than a short URL list, POST /v1/crawl starts an asynchronous task. It discovers pages through links and sitemaps, respects robots.txt and Crawl-delay, and can prioritize URLs and link text that match a query.

from serpkite import SerpKite

with SerpKite() as sk:
    task = sk.crawl(
        "https://docs.python.org/3/library/asyncio.html",
        include_paths=[r"^/3/library/asyncio"],
        query="task cancellation",
        limit=10,
        max_depth=2,
    )
    finished = sk.wait_for_crawl(task.id)

    if finished.status == "completed" and finished.result:
        for page in finished.result.pages:
            print(page.url, page.markdown)
    else:
        print(finished.status, finished.error)

A crawl can read up to 1,000 pages and follow up to 10 link hops. It reserves limit credits when queued, bills the pages read, and refunds the unused part. In the example, ten credits are reserved; if only four uncached pages are read, the final charge is four credits.

Poll GET /v1/crawl/{id}, or supply a webhook_url to receive a signed crawl.completed event. Results are kept for 24 hours, so save the content you need in your own application. You can cancel an unfinished task; a running crawl is still billed for the pages it has read. Details are in the crawl reference.

Watch for new results and page changes

Monitors save a search, news query or webpage URL and run it hourly, daily or weekly. Search and news monitors report results they have not seen before. Webpage monitors report changes to the page’s content. Every monitor keeps a run history; an optional webhook receives signed monitor.results events.

For example, create a daily news monitor:

curl https://api.serpkite.com/v1/monitors \
  -H "Authorization: Bearer $SERPKITE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "AI agent funding",
    "endpoint": "news",
    "q": "ai agents funding",
    "interval": "daily"
  }'

The first run is due immediately. Use the returned ID to read its history at GET /v1/monitors/{id}/runs, or manage it in the dashboard. Create a webpage monitor with endpoint: "webpage" and url in place of q to watch a changelog or pricing page.

Each run is billed like the search or page fetch it makes. A successful check still costs credits when it finds no new results or changes; failed requests and empty source results are free. Runs use the key that created the monitor, or the calling key when you update it or request a run. A dedicated key with a monthly limit helps keep recurring costs predictable. See monitors and spend controls.

Narrow a search before fetching content

You can now restrict results with include_domains, remove sources with exclude_domains, and set start_date or end_date. These work on search, news, images and videos. Search and news also support boost_domains, which moves matching results to the top without removing other sources.

On /v1/search, combine include_content with highlights: true to get up to three passages per fetched page instead of full page content:

{
  "q": "asyncio task cancellation",
  "include_domains": ["docs.python.org/3/library"],
  "include_content": 2,
  "highlights": true
}

The filters and passage ranking add no extra charge; fetching the pages has the usual include_content cost. Results also carry published_at when the displayed date can be parsed. Date ranges cannot be combined with time or tbs, and include/exclude domain filters cannot be combined with engine: "consensus". The search filters guide covers matching rules and supported combinations.

Use the new tools from an agent

The remote MCP server now includes map, extract, crawl and crawl_result alongside its search and webpage tools. An agent can discover a site’s pages, read relevant passages and collect a larger section through the same connection. Monitors are managed through REST, the SDKs or the dashboard.

Start with the MCP setup guide, the SDK reference, or the LangGraph tutorial. For every API change in this release, see the changelog.