RAG systems fail in specific, recognizable patterns — knowing them helps you diagnose a bad answer quickly rather than treating "the AI got it wrong" as a single, undifferentiated problem.
Common Failure Modes
| Failure Mode | Root Cause | Where to Look |
|---|---|---|
| Retrieval returns irrelevant chunks | Poor chunking, weak embedding model fit, or a query phrased very differently from source content | Chunking, Embedding Models |
| Correct chunk retrieved, but ranked too low to be included | top_k set too low, or embedding similarity alone insufficient for this content | Reranking |
| Answer spans multiple chunks, only one retrieved | Chunking split related information apart, insufficient overlap | Chunk Overlap |
| Model ignores retrieved context | Weak or missing grounding instructions in the prompt | RAG Prompt |
| Model fabricates when context is insufficient | No explicit "don't know" fallback instruction | RAG Prompt |
| Stale/outdated answers | Source content updated, but the vector database wasn't re-ingested | Document Processing |
| Exact terms/codes not found | Pure semantic retrieval missing exact-match content | Hybrid RAG |
A Practical Debugging Approach
1. Reproduce the bad answer with the exact same question.
2. Inspect what was actually retrieved — was the right content
even fetched? (retrieval problem, if not)
3. If retrieval was correct, inspect the assembled prompt sent
to the LLM — was the context clear and well-formatted?
4. If the prompt looks right, the issue is likely generation
behavior — check the grounding/fallback instructions.This mirrors the two-stage evaluation split in RAG Evaluation — isolate whether the failure is in retrieval or generation before trying to fix it.
Practical Use Case
Having this failure-mode table as a mental checklist dramatically speeds up debugging a production RAG issue compared to guessing — most "the AI gave a wrong answer" reports trace back to one of these well-known, specific causes.
Common Mistakes
- Jumping straight to "we need a better/bigger LLM" when the actual root cause is retrieval quality, chunking, or prompt design
- Not logging retrieved chunks alongside final answers, making post-hoc debugging of a specific bad response much harder
Interview Relevance
"Name three distinct ways a RAG system can fail, and how you'd detect each." — a strong answer pulls from distinct stages (retrieval, chunking, prompt/generation), not three variations of the same root cause.
Practice Question
A RAG system gives a confidently wrong answer about a policy that was updated last week. What's the most likely failure mode, and how would you confirm it?
