Website | GitHub | Docs | PyPi
mcp-server-webcrawl
Advanced search and retrieval for web crawler data. With mcp-server-webcrawl, your AI client filters and analyzes web content under your direction or autonomously. The server includes a fulltext search interface with boolean support, and resource filtering by type, HTTP status, and more.
mcp-server-webcrawl provides the LLM a complete menu with which to search, and works with a variety of web crawlers:
| Crawler/Format | Description | Platforms | Setup Guide |
|---|---|---|---|
| ArchiveBox | Web archiving tool | macOS/Linux | Setup Guide |
| HTTrack | GUI mirroring tool | macOS/Windows/Linux | Setup Guide |
| InterroBot | GUI crawler and analyzer | macOS/Windows/Linux | Setup Guide |
| Katana | CLI security-focused crawler | macOS/Windows/Linux | Setup Guide |
| SiteOne | GUI crawler and analyzer | macOS/Windows/Linux | Setup Guide |
| WARC | Standard web archive format | varies by client | Setup Guide |
| wget | CLI website mirroring tool | macOS/Linux | Setup Guide |
mcp-server-webcrawl is free and open source, and requires Claude Desktop and Python (>=3.10). It is installed on the command line, via pip install:
