Skip to main content

MCP Integration

Pry ships a Model Context Protocol (MCP) server so AI agents (Claude, Cursor, Hermes, and any MCP client) can scrape and extract the web natively — and pay via x402 automatically.

  • Server name: pry, version 3.0.0
  • Protocol version: 2024-11-05
  • API base URL: http://localhost:8002 (override with PRY_API_URL)
  • Enable with PRY_MCP_ENABLED=true (default)

Running the MCP server​

stdio transport (default, for local agents)​

pry mcp serve

Or directly: python -m mcp_production

HTTP + SSE transport (remote agents)​

mcp_sse.py implements the official MCP HTTP+SSE transport:

  1. Client connects to GET /mcp/sse
  2. Server sends an endpoint event with a POST URL (/mcp/messages/{session_id})
  3. Client POSTs JSON-RPC messages to that URL
  4. Server replies by enqueueing JSON-RPC responses as message events on the SSE stream

Sessions are kept in memory (5-minute idle TTL). For production multi-worker deployments, use a Redis-backed queue or sticky sessions.

Claude Desktop / Cursor configuration​

{
  "mcpServers": {
    "pry": {
      "command": "python",
      "args": ["-m", "mcp_production"]
    }
  }
}

For Hermes or any stdio MCP client, the equivalent config is command: "pry", args: ["mcp", "serve"].

Tools​

The MCP server exposes these tools:

ToolWhat it does
pry_scrapeScrape any URL to clean markdown (POST /v1/scrape)
pry_crawlCrawl a site up to N pages (POST /v1/crawl)
pry_extractExtract structured data with CSS selectors or a JSON schema (POST /v1/extract/css)
pry_templateExecute a pre-built scraper template
pry_search_templatesSearch the 110+ scraper templates
pry_monitorCreate a scheduled page monitor
pry_complianceRun a GDPR/compliance check on a URL
pry_enrichEnrich a page (tech stack, metadata)
pry_parse_documentParse PDF/DOCX/OCR documents (POST /v1/parse)
pry_screenshotTake a screenshot of a URL
pry_x402_pricingQuery x402 pay-per-call pricing
pry_referralsReferral tracking

Example tool call (JSON-RPC):

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "pry_scrape",
    "arguments": {
      "url": "https://example.com",
      "formats": ["markdown"]
    }
  }
}

Legacy HTTP bridge​

The repo's README references an HTTP bridge (POST /mcp/call with {"name": "pry_scrape", "arguments": {...}} and GET /mcp/tools). These endpoints exist only in mcp_production.py, which is not deployed — use pry mcp serve (stdio) or the HTTP+SSE server (mcp_sse.py) for MCP access. GET /v1/ai/mcp-config also serves the current MCP server config.

Agent onboarding SKILL.md​

When you give an agent access to Pry, provide a skill file so it knows how to use the tools correctly. A minimal onboarding SKILL.md:

# Pry — Web Scraping & Browser Automation

Use the `pry_*` MCP tools to scrape and extract web content.

## Rules
- Prefer `pry_extract` with CSS selectors for structured data (deterministic, free).
- Use `pry_scrape` for markdown content; add `bypassCloudflare: true` for WAF-protected sites.
- Use `pry_parse_document` for PDF/DOCX/OCR documents.
- Use `pry_crawl` (maxPages ≤ 10) before reaching for `pry_scrape` on multi-page sites.
- For AI extraction of messy pages, use `pry_extract` with an instruction + schema.
- Check `pry_x402_pricing` before calling paid operations when x402 is enabled.
- Never scrape login-walled content without authorization; respect robots.txt and ToS.

## Example
1. `pry_scrape` https://example.com → markdown
2. `pry_extract` with schema `{"name": "string", "price": "number"}` → JSON

x402 + MCP​

When PRY_X402_ENABLED=true, paid MCP operations go through the x402 flow — agents pay per call from their wallet without an account. See x402 Pay-per-call for the full flow.

Next steps​