Extracting usable content from HTML (web pages) for RAG means separating actual content from navigation, ads, and boilerplate — an HTML page's DOM structure is a mix of meaningful content and page-scaffolding that a naive "grab all text" approach doesn't distinguish.
The Core Problem
A typical web page's raw text includes things like:
"Home | Products | About | Contact | Sign In
[actual article content...]
Related Articles: ... Subscribe to our newsletter ...
© 2026 Company Name. All rights reserved. Privacy | Terms"
Naive extraction grabs ALL of this — including navigation menus,
footers, and unrelated "related content" links — mixed in with
the actual article text you actually want.Content Extraction Strategies
| Approach | How It Works |
|---|---|
| Semantic HTML targeting | Prioritize content inside <article>, <main> tags where present — see Semantic HTML for why well-structured pages make this easier |
| Readability-style extraction | Heuristic algorithms that identify the "main content" block of a page by analyzing text density and DOM structure, filtering out likely navigation/ads/boilerplate |
| Site-specific selectors | For a known, controlled set of pages, manually targeting the specific CSS selector containing real content — more precise, less generalizable |
A Simplified Example
def extract_html_content(html):
soup = html_parser(html)
# Prefer semantic content containers if present
main_content = soup.find("article") or soup.find("main")
if main_content:
return clean_text(main_content.get_text())
# Fallback: heuristic content extraction
return heuristic_extract_main_content(soup)Practical Use Case
Ingesting a company's own documentation site, a competitor's public product pages for research, or scraped web content for a knowledge base are all common HTML-for-RAG scenarios — extraction quality directly determines whether retrieved chunks are genuinely useful content or navigation-menu noise.
Common Mistakes
- Extracting all visible text from a page without filtering navigation, ads, and footers — resulting in noisy, low-quality chunks
- Not handling pages with heavy client-side JavaScript rendering, where the raw HTML fetched doesn't contain the actual rendered content at all
- Assuming a generic extraction heuristic works equally well across structurally very different websites
Interview Relevance
"Why is extracting content from a web page harder than extracting from a plain text file?" — HTML mixes genuine content with navigation, ads, and boilerplate in the same document, requiring deliberate filtering that plain text doesn't need.
Practice Question
A RAG system ingesting a company blog is retrieving chunks that are mostly navigation menu text. Propose a specific fix.
