Extraction API

Turn raw scraped content into structured data, and parse documents.

POST /v1/extract/css

Extract structured JSON from a URL using a CSS selector schema — no LLM required, fully deterministic and free.

Request body:

Field	Type	Default	Description
url	string	—	Required. URL to extract from
schema	object	—	Required. CSS selector schema (see below)
bypass_cloudflare	boolean	true	Attempt Cloudflare bypass

Schema format:

{
  "name": "products",
  "base_selector": ".product-card",
  "fields": [
    {"name": "title", "selector": "h3", "type": "text"},
    {"name": "price", "selector": ".price", "type": "text", "transform": "float"},
    {"name": "link", "selector": "a", "type": "attribute", "attribute": "href"},
    {"name": "in_stock", "selector": ".stock", "type": "exists"}
  ]
}

Field type options include text, attribute, exists, html, and transform values like float/int for normalization. When base_selector is set, each matched element yields one record with the fields applied relative to it.

Example:

curl -X POST http://localhost:8005/v1/extract/css \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <key>" \
  -d '{
    "url": "https://store.com/products",
    "schema": {
      "name": "products",
      "base_selector": ".product-card",
      "fields": [
        {"name": "title", "selector": "h3", "type": "text"},
        {"name": "price", "selector": ".price", "type": "text", "transform": "float"}
      ]
    }
  }'

Response (200):

{
  "success": true,
  "data": {
    "products": [
      {"title": "Widget", "price": 29.99},
      {"title": "Gadget", "price": 49.5}
    ]
  }
}

POST /v1/extract/llm

Extract structured data using an LLM with intelligent chunking. Chunks content by strategy, optionally filters by relevance to a query, then extracts from each chunk.

Request body:

Field	Type	Default	Description
url	string	—	Required. URL to extract from
instruction	string	Extract all key information from this content.	Extraction instruction
schema	object	—	Desired output JSON schema
chunk_strategy	string	topic	Chunking strategy (topic, sentence, regex, …)
query	string	—	Relevance filter query
top_k	integer	5	Max chunks to extract from

Example:

curl -X POST http://localhost:8005/v1/extract/llm \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <key>" \
  -d '{
    "url": "https://store.com/product/123",
    "instruction": "Extract the product name, price, and availability",
    "schema": {"name": "string", "price": "number", "availability": "string"},
    "chunk_strategy": "topic"
  }'

Response (200): structured JSON matching the requested schema.

POST /v1/parse

Parse a document (PDF, DOCX, image, CSV, JSON) to text.

Request body (ParseRequest):

Field	Type	Default	Description
url	string	—	Required. Document URL
timeout	integer	60	Timeout in seconds

curl -X POST http://localhost:8005/v1/parse \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <key>" \
  -d '{"url": "https://example.com/document.pdf", "timeout": 60}'

Response (200): { "text": "…extracted text…", "format": "pdf" }

Formats supported: PDF, DOCX, images (OCR), CSV, JSON.

Related parsing endpoints:

Endpoint	Purpose
POST /v1/pdf/extract	PDF table extraction (pdf_url, method — default pdfplumber)
POST /v1/ocr/extract	Image OCR
POST /v1/schema/extract	Schema.org / JSON-LD / microdata extraction
POST /v1/schema/extract-html	Schema extraction from raw HTML
POST /v1/extract-table	Table extraction

POST /v1/shadow-dom

Scrape a page and extract content from Shadow DOM components. Useful for modern web apps built with Lit, web components, or frameworks that use Shadow DOM encapsulation.

Request body:

Field	Type	Default	Description
url	string	—	Required. URL with Shadow DOM content
flatten	boolean	true	Flatten nested shadow roots

curl -X POST http://localhost:8005/v1/shadow-dom \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <key>" \
  -d '{"url": "https://app.example.com", "flatten": true}'

Response (200): extracted text/content from shadow roots, with flatten merging nested roots into one tree.

POST /v1/capture/lazy

Detect and handle lazy-loaded content and infinite-scroll patterns. Optionally auto-scroll to load all content before extraction.

Request body:

Field	Type	Default	Description
url	string	—	Required. URL
auto_scroll	boolean	true	Auto-scroll to trigger lazy load
max_scrolls	integer	5	Max scroll iterations

POST /v1/capture/network

Extract API calls, GraphQL queries, and network patterns from a page. Useful for understanding how SPAs load data and for finding hidden API endpoints.

Request body: { "url": "https://app.example.com" }

Response (200): list of network requests (URLs, methods, query types).

Other extraction endpoints

Endpoint	Purpose
POST /v1/extract	Generic extraction with JSON schema + optional AI fallback
POST /v1/extract/fields	Structured field extraction
POST /v1/suggest	AI field suggestion for a URL
POST /v1/emails	Email address extraction
POST /v1/links	Link analysis
POST /v1/seo/analyze	SEO analysis (title, meta, headings, keywords, readability)

Next steps

• Automation API — browser automation and sessions
• Scraping API — raw scrape/crawl/batch