webcrawl-mcp
A lightweight MCP server that gives Claude Code (or any MCP client) the ability to scrape, search, map, and crawl the web — using free, open-source libraries. Firecrawl is supported as an optional fallback for JS-heavy sites when you have a key.
Why
Most scraping doesn't actually need a headless browser. trafilatura handles the ~80% case (articles, docs, blogs) locally, which is faster and keeps external API usage to a minimum. This server routes the easy stuff through local extraction and only falls back to Firecrawl when content quality is genuinely poor.
Tools
| Tool | Purpose |
|---|---|
webcrawl_scrape |
Fetch a single URL → {content, source} (markdown + provenance) |
webcrawl_search |
DuckDuckGo search (optionally scrape results, each with provenance) |
webcrawl_map |
Discover same-domain URLs from a starting page |
webcrawl_crawl |
BFS crawl multiple pages (each result includes provenance) |
The source field on scraped content is one of static_http, static_http_retry, firecrawl_transport_fallback, or firecrawl_quality_fallback — see Fallback behavior.
