Webcrawl Anything

Crawl, search, and extract structured evidence from any website, SPA, PDF, Office file, YouTube video, GitHub repo, ArXiv paper, shopping offer, travel fare, or CVE advisory — from a single CLI and MCP server.

What You Can Crawl & Search

Run npx webcrawl targets at any time to inspect all registered targets from your terminal.

Quickstart

One command registers the @webcrawl/mcp server and installs the webcrawl-anything Agent Skill (SKILL.md) for Claude Code, Cursor, and Codex:

npx webcrawl setup

Or target specific coding agents:

npx webcrawl setup --agent claude --agent cursor --agent codex --agent gemini --agent windsurf

Check your local engine, stealth browser, and parser readiness:

npx webcrawl doctor

Zero-Key CLI Usage

Webcrawl Anything runs out-of-the-box locally with zero API keys using the built-in impit TLS-impersonated fetcher, document/media parsers, BM25 + semantic ranker, and vertical extractors:

# Scrape any website, GitHub repo, ArXiv paper, PDF, or YouTube video into clean Markdown
npx webcrawl scrape https://github.com/torvalds/linux
npx webcrawl scrape https://arxiv.org/abs/1706.03762
npx webcrawl scrape https://www.youtube.com/watch?v=dQw4w9WgXcQ

# Search the live web or structured verticals
npx webcrawl search "react 19 compiler" -n 5
npx webcrawl search "sony wh-1000xm5" -c shopping
npx webcrawl search "boutique hotels in shibuya" -c hotels
npx webcrawl search "3 bedroom house in marrickville" -c real-estate --sub buy
npx webcrawl search "CVE-2024-3094 xz utils" -c security --sub cve

# Discover every URL on a domain (sitemap.xml + robots.txt + link graph)
npx webcrawl map https://docs.stripe.com --filter webhook -n 50

# Crawl up to 25 pages in parallel with target evidence
npx webcrawl crawl https://docs.stripe.com/webhooks -n 5

# Extract structured JSON from any page
npx webcrawl extract https://news.ycombinator.com --prompt "top 5 stories with points"

Example Prompts (Once Connected to Your Agent)

Once npx webcrawl setup has registered the MCP server and skill, ask Claude Code, Cursor, or Codex directly:

• "Search for CVE-2024-3094 in the security category and show me the CVSS score, CISA KEV status, and fixed versions."
• "Scrape https://arxiv.org/abs/2305.18290 and summarize the DPO objective and experimental results."
• "Compare retail prices and stock availability for the Apple Mac Mini M4 across US retailers using web_search with category: shopping."
• "Map https://modelcontextprotocol.io, pick the specification pages, and crawl them into Markdown."
• "Parse the PDF at https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf and verify whether any pages required OCR."

Local Engine vs. webcrawl.app Cloud

Webcrawl Anything uses the exact same TypeScript engine (src/) locally and in the hosted webcrawl.app cloud API. You can start with zero keys locally and upgrade to cloud with one command whenever you need managed stealth browsers or sub-50ms global cache hits:

To connect your CLI and MCP server to webcrawl.app cloud:

npx webcrawl login
npx webcrawl whoami

Compact MCP Schema Profile for Context-Constrained Agents

By default, npx webcrawl setup configures WEBCRAWL_MCP_SCHEMA_PROFILE=compact so MCP tool schemas use minimal token overhead inside Claude Code, Cursor, and local LLM context windows while keeping full server-side validation.

To switch to verbose tool parameter descriptions:

npx webcrawl setup --schema-profile full

Architecture & Shared Engine Flywheel

webcrawl/
├── src/
│   ├── targets/        # Pluggable Target & Extractor Registry (web, pdf-office, youtube, github, arxiv, llms-txt, shopping, travel, real-estate, cve)
│   ├── engine/         # Multi-tier fetcher (impit TLS impersonation → stealth browser → HTML-to-Markdown + SHA-256 cache)
│   ├── parse/          # Native PDF inspector + OCR fallback, DOCX/XLSX/PPTX parser, YouTube caption extractor
│   ├── extract/        # Vertical structured extractors (shopping, flights, hotels, real-estate, business, CVE/NVD/OSV)
│   ├── search/         # Multi-provider search fan-out + vertical pipelines
│   ├── ranker/         # Hybrid BM25 + semantic vector + RRF + MMR diversity reranker
│   └── local-engine.ts # In-process zero-key backend powering CLI & MCP locally
├── packages/
│   ├── cli/            # `webcrawl` CLI (`npx webcrawl search|scrape|map|crawl|extract|rerank|targets|setup|doctor`)
│   ├── mcp/            # `@webcrawl/mcp` stdio & Streamable HTTP MCP server
│   └── client/         # `@webcrawl/client` zero-dependency TypeScript SDK
└── api/
    └── index.ts        # Production `webcrawl.app` API powered by the exact same `src/` engine

Every community improvement to a target plugin (src/targets/), vertical extractor (src/extract/), or document parser (src/parse/) immediately powers both local CLI/MCP users and the hosted webcrawl.app platform.

Contributing a New Target or Extractor

Want to add a new target (e.g., Reddit threads, Hacker News discussions, SEC EDGAR filings, npm/PyPI packages, or a regional real-estate/shopping portal)?

See CONTRIBUTING.md for the 3-step guide to registering a CrawlTargetPlugin in src/targets/ with a fixture test in src/__tests__/targets.test.ts.

License

MIT