Convert Scraped HTML to Markdown for AI Pipelines

Web scraping gives you raw HTML — complete with navigation, ads, cookie consent dialogs, and layout markup that your AI pipeline doesn't need. Manually cleaning scraped pages doesn't scale when you're processing hundreds or thousands of pages.

This converter uses a density-based content extraction algorithm to automatically identify the "real content" on each page. Navigation bars, footers, sidebars, and promotional blocks are stripped. What remains is converted to clean GitHub-Flavored Markdown.

The Problem with Raw Scraped HTML

Noise ratio — A typical web page is 70-80% boilerplate; only 20-30% is actual content
Inconsistent structure — Every site uses different layouts, making regex-based extraction brittle
Token waste — Feeding raw HTML to an LLM burns tokens on <nav>, <footer>, and inline styles
Embedding quality — Vector embeddings of noisy HTML produce poor retrieval results

How It Works

Scrape your target — Use wget, Scrapy, Puppeteer, or any crawler to download pages as .html files
ZIP the output — Bundle the scraped HTML into a ZIP archive (up to 200 files, 100 MB)
Convert in-browser — The WASM engine scores each container element by text density, selects the content block, and outputs GFM Markdown
Download results — Get clean .md files preserving the original directory structure

Works With Any Scraper

wget -r — recursive site mirroring
Scrapy — Python crawling framework
Puppeteer / Playwright — headless browser scraping (save as HTML)
curl + scripts — custom URL list downloads
HTTrack — website copier

Ideal For

Building RAG (Retrieval-Augmented Generation) knowledge bases from public documentation
Creating training datasets for domain-specific models
Migrating web content to Markdown-based systems (Obsidian, Notion, MkDocs)
Archiving websites in a readable, searchable format

Privacy Guarantee

All processing runs locally in your browser using WebAssembly. Your scraped content is never uploaded to any server — making this safe for proprietary data, internal wikis, and sensitive material.