Raw extracted text — regardless of source format — usually needs cleaning before chunking and embedding: removing artifacts, fixing extraction errors, and stripping content that adds noise rather than meaning.
Common Cleaning Steps
| Issue | Example | Fix |
|---|---|---|
| Repeated headers/footers | "Page 4 of 52 | Confidential" on every page | Detect and strip repeated boilerplate patterns |
| Excessive whitespace/line breaks | Irregular spacing from PDF extraction | Normalize whitespace |
| OCR artifacts | Misrecognized characters, broken words | Spell-check/correction pass, or flag for manual review if error rate is high |
| Navigation/boilerplate text (from HTML) | "Home | About | Contact" menu text | Content-extraction filtering (see HTML for RAG) |
| Encoding issues | Garbled special characters (e.g. "don’t" instead of "don't") | Proper character encoding detection/normalization |
A Basic Cleaning Function
import re
def clean_text(raw_text):
text = re.sub(r'\s+', ' ', raw_text) # normalize whitespace
text = remove_repeated_boilerplate(text) # strip repeated
# headers/footers
text = fix_encoding_issues(text)
return text.strip()Cleaning Is a Real Quality Investment, Not Busywork
Unclean text doesn't just look messy — it directly degrades embedding quality (noisy input produces less precise vectors) and can waste chunk space on content that adds no value. Time spent on cleaning, especially for messy source formats like scanned PDFs, is time well spent relative to its impact on final retrieval quality.
Practical Use Case
Ingesting a large archive of historical scanned documents, or web-scraped content from many different sites, typically requires more aggressive, format-specific cleaning than ingesting well-authored Markdown documentation — the cleaning effort should scale with how messy the source format tends to be.
Common Mistakes
- Skipping a cleaning step entirely, embedding raw extracted text (including boilerplate and artifacts) directly
- Over-aggressive cleaning that accidentally strips meaningful content along with noise (e.g. removing all short lines, which might delete genuinely short but meaningful headings)
- Not testing cleaning logic against a representative sample before running it across an entire large document collection
Interview Relevance
"Why does document cleaning matter for RAG, beyond just making text 'look nicer'?" — noisy text directly degrades embedding quality and wastes limited chunk/context space on non-content, both of which measurably hurt retrieval and answer quality.
Practice Question
Write a cleaning function that removes a repeated footer pattern like "Confidential — Page X of Y" that appears at the end of every extracted page.
