FAQ
Is scraping legal / ethical?
Scraping lives in a gray zone that depends on where you operate and what you scrape. Pry provides compliance tooling to help you stay on the right side:
POST /v1/compliance/check— full GDPR/CCPA compliance check on a URLPOST /v1/gdpr/*— consent recording, data deletion requests, retention policies, data portability, audit logPOST /v1/training/clean— PII/copyright stripping for training datasets
Good practice: respect robots.txt, rate-limit politely (Pry defaults to 120
RPM per IP), avoid login-walled content, and don't scrape personal data
without a lawful basis. The compliance endpoints exist precisely because
Pry's own build is GDPR-conscious.
Does Pry bypass Cloudflare?
Yes — automatically. The 10-tier fallback chain walks
direct → cloudscraper → FlareSolverr → undetected-chromedriver → Playwright →
Googlebot → Tor → archive/cache fallbacks and returns the first successful
result. Cloudflare/WAF challenges are handled by the FlareSolverr sidecar
(Docker image, health-checked by compose). You can disable it per-request with
"bypassCloudflare": false.
There is no guaranteed bypass — aggressive WAFs (e.g. DataDome on some
sites) may still win. Use POST /v1/detect-block to see which vendor is
blocking you and tune from there.
Self-hosted vs hosted — which should I use?
| Self-hosted | Hosted | |
|---|---|---|
| Cost | Free (MIT core); infrastructure is yours | Pro $49/mo, Team $199/mo |
| Privacy | All data stays on your machines | Data flows through our service |
| Ops | You run Docker/nginx/updates | Zero ops |
| Stealth module (BSL) | Free non-prod; commercial license for prod | Included |
| Best for | Privacy-sensitive teams, devs, tinkerers | Teams that don't want ops |
You can also run self-hosted and enable x402 pay-per-call to charge your own users per call — the best of both.
How does Pry scale?
Pry is a single FastAPI app + FlareSolverr + optional Redis/Postgres. To scale:
- Redis (
PRY_REDIS_URL) — shared cache backend with TTL invalidation. - Postgres (
PRY_DATABASE_URL) — async SQLAlchemy for stateful features. - Async job queue (
jobqueue.py) + WebSocket streaming for long crawls. - Horizontal: run multiple API replicas behind the load balancer; sessions and job state are the stateful parts to share (Redis-backed MCP SSE sessions are recommended for multi-worker deployments).
Resource limits in the default compose: 2 GB / 2 CPUs for Pry, 512 MB / 1 CPU for FlareSolverr. Playwright scraping is the memory-hungry part — budget accordingly.
How does Pry compare to Crawl4AI / Firecrawl / Scrapy?
| Pry | Crawl4AI | Firecrawl | Scrapy | |
|---|---|---|---|---|
| License | MIT + BSL stealth | MIT | Commercial SaaS (open-core) | BSD |
| Self-hostable | ✅ | ✅ | Limited | Framework only |
| Cloudflare bypass | ✅ 10-tier automatic | Partial | ✅ (hosted) | ❌ (DIY) |
| Browser automation | ✅ sessions, stealth, Camoufox | ⚠️ via Playwright | ⚠️ limited | ❌ (separate tools) |
| Document parsing | ✅ PDF/DOCX/OCR/CSV/JSON | ⚠️ | ✅ | ❌ |
| LLM extraction | ✅ chunked | ✅ | ✅ | ❌ |
| x402 crypto pay-per-call | ✅ native | ❌ | ❌ | ❌ |
| Compliance/GDPR tooling | ✅ built-in | ❌ | ⚠️ | ❌ |
| Commerce/CRM sync | ✅ WooCommerce, Shopify, Salesforce, HubSpot | ❌ | ❌ | ❌ |
Pry positions itself as a Firecrawl + Crawl4AI + Browserless replacement in one self-hosted app — with x402 as the differentiator: AI agents pay per call without an account, at ~$0.001–$0.10 per operation.
What does "no API keys needed" mean?
For the self-hosted lane: run Pry locally and use it without registering
anywhere. There is no hosted account to create. Once you expose the API beyond
loopback, set PRY_API_KEY (fail-closed auth) — that's your own key, not
Pry's.
Does Pry store my data?
Self-hosted: all data lives under PRY_DATA_DIR (default ~/.pry/) on your
machine — quality history, monitors, sessions, encrypted credential vault,
GDPR records. Nothing leaves your host unless you configure destinations
(webhooks, S3, GCS, SFTP) or the hosted lane.
Can I charge users with Pry?
Yes — that's the x402 pay-per-call lane. Enable PRY_X402_ENABLED=true,
point PRY_X402_PAY_TO at your wallet, and every scrape/crawl/extract earns
micropayments (USDC/USDT). See x402 Pay-per-call.
Is the stealth module free?
Free for personal, non-production, academic, and non-commercial use (BSL 1.1 Additional Use Grant) plus a 90-day evaluation. Production use requires a commercial license from enterprise@rugmunch.io. Converts to MIT on 2029-01-01.
Where is the source?
git.rugmunch.io/RugMunchMedia/pryscraper — mirrored to Codeberg and GitLab.
How do I report a security issue?
Email security@rugmunch.io (PGP key available on request). Response within 24 hours; critical patches within 7 days.
Next steps
- Troubleshooting — common errors and fixes
- Pricing & Monetization — the three lanes