MarkCrawl by iD8 🕷️📝
Turn any webpage or website into clean Markdown for LLM pipelines — in one command.
Latest: v0.11.1 (2026-05-12) — default aggregator URL filter. See What's New below.
pip install markcrawl
markcrawl --base https://docs.example.com --out ./output --show-progressMarkCrawl is a crawl-and-structure engine. It fetches one page or crawls an entire website, strips navigation/scripts/boilerplate, and writes clean Markdown files with a structured JSONL index. Every page includes a citation with the access date. No API keys needed.
Everything else — LLM extraction, Supabase upload, MCP server, LangChain tools — is optional and installed separately.
Want a hosted API instead of running locally? Join the waitlist — we're gauging interest.
LLM agents: Load docs/LLM_PROMPT.md as a system prompt to generate correct MarkCrawl commands automatically.
What's New
Install or upgrade with pip:
pip install --upgrade markcrawl
pip show markcrawl | grep Version # confirm the installed version
markcrawl --help | head -1 # confirm the binary on $PATH is the upgraded one
