Clean HTML for AI Training Data
Building a fine-tuning dataset or training corpus from web content? The biggest challenge isn't getting the HTML — it's cleaning it. Every web page comes wrapped in navigation, ads, cookie dialogs, related-article widgets, and layout markup that pollutes your training data.
This tool applies a density-based content extraction algorithm (the same approach used by academic boilerplate removal research) to automatically identify and extract the primary content from each page. The result is clean, structured Markdown that represents what a human would consider "the article."
Why Clean Data Matters for AI Training
- Quality in, quality out — Models trained on noisy data learn to reproduce noise (navigation text, cookie consent phrases, "click here" patterns)
- Deduplication — Boilerplate text (headers, footers, nav) is identical across thousands of pages, creating massive unintentional duplicates in your corpus
- Token efficiency — Training on content-only text means every token contributes to learning the domain, not the page chrome
- Better evaluation — Clean test sets give you honest metrics about model performance on actual content
The Density Extraction Algorithm
The converter scores each container element (<div>, <section>,<article>, <main>) by computing a signal score based on:
- Text density — Ratio of visible text to HTML markup within the container
- Structural signals — Presence of headings, paragraphs, and semantic elements
- Noise indicators — Navigation links, form elements, and repetitive patterns typical of boilerplate
The highest-scoring container is selected as the content block. Everything else — nav, footer, sidebar, ads — is discarded before Markdown conversion.
How It Works
- Collect your HTML — Crawl, scrape, or export the pages you want in your training set
- Batch convert — ZIP up to 200 HTML files and upload them (up to 100 MB per archive)
- Download clean Markdown — Each file is processed independently, so one bad page doesn't block the rest
- Use in your pipeline — Feed the
.mdfiles into your tokenizer, chunker, or fine-tuning script
Compared to Other Approaches
| Approach | Pros | Cons |
|---|---|---|
| Regex stripping | Fast | Brittle, site-specific, misses semantic structure |
| BeautifulSoup + heuristics | Flexible | Requires per-site tuning, slow on large batches |
| Readability.js | Good for articles | Designed for single pages, not batch processing |
| This tool (density extraction) | Works on any site without tuning, batch processing, GFM output | May not handle very unusual layouts |
Best Practices for Training Data
- Convert in batches by domain/topic for thematic consistency
- Review a random sample of outputs before feeding to your training pipeline
- Filter out empty results (pages with no extractable content) — these indicate non-article pages like index/listing pages
- Combine with metadata (source URL, date, category) for richer datasets
Privacy
Processing happens entirely in your browser via WebAssembly. Your training data is never uploaded to any server. This is especially important when working with proprietary content, licensed datasets, or data subject to agreements.