Extract structured data from any web page with JSON Schema

The classic way to get data out of web pages is a parser per site: CSS selectors or XPath for the price, the plan name, the author, the date. It works until the site changes its markup, and then it fails quietly, returning empty strings that look like data. Multiply that by a few hundred sites and maintaining parsers becomes the job.

A language model reading the page doesn’t care about the markup. Describe the fields you want as a JSON Schema, give it the page’s content, and ask for JSON that matches the schema. That is what output_schema on SerpKite’s /v1/extract does: you send up to 20 URLs and a schema, and each page comes back as JSON, written by Claude Haiku 5.5 from that page’s content.

This tutorial covers a first extraction, schema design that avoids made-up values, classification, running it over a whole site, and what it costs.

How it works

1. SerpKite fetches each URL (HTML or PDF), logged out and through its proxies, rendering JavaScript pages when needed.
2. The page is converted to clean Markdown, without navigation, ads and boilerplate.
3. The model fills your schema from that Markdown, guided by your optional prompt.
4. Each result carries the page’s markdown and the extracted output; pages that couldn’t be read are listed in failed with a reason.

Values come only from the page. Instructions written inside a page are ignored, so a page can’t tell the model to return something else.

A first extraction in Python

The Python SDK accepts a pydantic model as the schema, so the same class describes the request and validates the result:

pip install -U serpkite
export SERPKITE_API_KEY=skt_live_...

from typing import Optional

from pydantic import BaseModel, Field
from serpkite import SerpKite


class Plan(BaseModel):
    name: str
    price_per_month: Optional[float] = Field(
        None, description="List price in USD per month; null when the plan has no list price"
    )
    seats: Optional[int] = Field(None, description="Included seats; null when not stated")


class PricingPage(BaseModel):
    plans: list[Plan]
    free_trial: bool


with SerpKite() as sk:
    res = sk.extract(
        ["https://www.example.com/pricing"],
        output_schema=PricingPage,
        prompt="Monthly prices in USD. Use null when a value is not on the page.",
    )

for page in res.results:
    if page.output_error:
        print("Extraction failed:", page.url, page.output_error)
        continue
    pricing = PricingPage.model_validate(page.output)
    for plan in pricing.plans:
        print(plan.name, plan.price_per_month)

for failure in res.failed:
    print("Could not read:", failure.url, failure.error.code)

The output for a typical pricing page looks like this:

{
  "plans": [
    { "name": "Starter", "price_per_month": 19, "seats": 1 },
    { "name": "Team", "price_per_month": 79, "seats": 5 },
    { "name": "Enterprise", "price_per_month": null, "seats": null }
  ],
  "free_trial": true
}

The “Enterprise” plan has no list price, so it comes back as null. That is the schema doing its job, not a failure.

The same call in TypeScript

The TypeScript SDK takes a plain JSON Schema object. output is typed unknown, so cast it to your own type or validate it with zod:

import { SerpKite } from "serpkite";

type Article = { headline: string; author: string | null; published: string | null; topics: string[] };

const sk = new SerpKite(); // reads SERPKITE_API_KEY

const res = await sk.extract({
  urls: [
    "https://www.example.com/blog/launch",
    "https://www.example.com/blog/pricing-update",
  ],
  output_schema: {
    type: "object",
    properties: {
      headline: { type: "string" },
      author: { type: ["string", "null"] },
      published: { type: ["string", "null"], description: "YYYY-MM-DD" },
      topics: { type: "array", items: { type: "string" }, maxItems: 5 },
    },
    required: ["headline", "topics"],
  },
  prompt: "Dates as YYYY-MM-DD. Topics are short lowercase phrases.",
});

for (const page of res.results) {
  if (page.output_error) continue;
  const article = page.output as Article;
  console.log(page.url, article.headline, article.published);
}
console.log(res.meta.credits_used, "credits");

With curl, the body is the same JSON: urls, output_schema and prompt. The response is the usual extract response (request, results, failed, meta), and meta.llm reports the provider, model and tokens used.

Designing a schema that doesn’t invent values

The model fills every field you ask for, so the schema should make “not on the page” an allowed answer.

• Make optional facts nullable. "type": ["number", "null"] plus “null when missing” in the prompt or the field’s description. Without it, a missing price may be filled with something plausible.
• Use description for format. “YYYY-MM-DD”, “USD per month”, “ISO 3166 country code”. Field descriptions are part of what the model reads.
• Use enums for categories. "enum": ["monthly", "yearly", "one-time"] gives you values you can group on, instead of free text.
• Put normalisation rules in prompt. Currency conversion, units, which of several prices counts. The prompt takes up to 4,096 characters; the schema up to 16 KB.
• Keep one schema per page type. A pricing schema on a blog post returns mostly nulls. If your URLs are mixed, classify them first (next section).

Classification: yes/no and enum schemas

A schema whose fields are all booleans or enums is a classification: “is this a pricing page?”, “which category is this product?”. Those go to OpenAI's Decisions API instead of Claude, which can only answer with values the schema allows.

{
  "urls": [
    "https://www.example.com/blog/launch",
    "https://www.example.com/pricing",
    "https://www.example.com/login"
  ],
  "output_schema": {
    "type": "object",
    "properties": {
      "page_type": { "enum": ["pricing", "docs", "blog", "login", "other"] },
      "mentions_free_tier": { "type": "boolean" }
    }
  }
}

This makes a cheap first pass over a list of URLs: classify them, then run the detailed schema only on the pages of the type you want.

Summaries next to the JSON

summary: true adds a 3 to 6 sentence summary of each page in results[].summary, focused on query when you send one. It combines with output_schema and with highlights, so one call can return a page’s data, a summary for a human, and the passages most relevant to a question.

Extracting from a whole site

/v1/extract needs URLs. When you only have a domain, list its pages with /v1/map, which reads the sitemaps and start page and can rank the URLs by a search phrase. Then extract in batches of 20:

from serpkite import SerpKite

with SerpKite() as sk:
    found = sk.map("https://www.example.com/", search="pricing plans", limit=60)
    urls = [item.url for item in found.results]

    rows = []
    for i in range(0, len(urls), 20):
        res = sk.extract(urls[i : i + 20], output_schema=PricingPage)
        rows += [(page.url, page.output) for page in res.results if page.output is not None]

print(len(rows), "pages extracted")

For a documentation site or a catalog with thousands of pages, start an async crawl to collect the Markdown, then extract the pages you need. The site ingestion guide covers map, crawl and extract together.

What it costs

What	Credits per page
Reading the page	1 (0.5 from cache with max_age)
output_schema	+4
summary: true	+4

A page with a schema extraction costs 5 credits. Per 1,000 pages that is $5.00 → $3.00, from the smallest credit pack to the best-value one. What you don’t pay for:

• A URL that can’t be read: it is listed in failed and costs nothing.
• A page that turns out to be a bot check, a login wall, a soft 404 or an empty app shell: also listed in failed, also free.
• An extraction that fails: reported in output_error (or summary_error), and you pay only for the page.

With summary or output_schema, the call’s timeout defaults to 75 seconds, and the last 25 are reserved for the model. A page still loading at the deadline is reported as failed and not billed.

When you don’t have the URLs: research instead

Extraction assumes you know which pages hold the data. When you don’t (“find the managed Postgres providers with an EU region and their starting price”), the work is search first, then reading, then extraction. /v1/research does that as a background task: it plans Google searches, reads and ranks the pages, and writes a cited report. Pass the same kind of output_schema and the findings come back as JSON in result.output, next to the report and its sources. It costs a flat 30, 75 or 200 credits per task (lite, standard, deep), charged only when the report is written. See the deep research guide.

Privacy

Page text and your prompt are sent to the model provider (Anthropic, or OpenAI for classification schemas) under API terms that exclude training. SerpKite doesn’t log them, and extractions are not cached. Details are in privacy and data retention.

Further reading

• Structured extraction guide and the extract reference
• AI Data Extraction API and structured data extraction
• SerpKite vs Firecrawl, if you are comparing extraction APIs
• Cited AI answers, JSON extraction and research in one API, the launch post