RAG substantially reduces hallucination risk compared to ungrounded generation, but doesn't eliminate it — a model can still ignore, misread, or overgeneralize from correctly-retrieved context.
How Hallucination Still Happens Even With RAG
| Failure Pattern | Example |
|---|---|
| Ignoring retrieved context entirely | Model answers from its own memorized (possibly outdated) knowledge instead of the provided context |
| Misreading the context | Context says "14 days for damaged items, 30 days for standard returns" — model conflates the two and states 30 days for damaged items |
| Overgeneralizing from partial context | Context covers one specific product's warranty; model applies the same terms to a different, unmentioned product |
| Fabricating despite insufficient context | No explicit "don't know" instruction, so the model guesses a plausible-sounding answer rather than admitting the context doesn't cover it |
Detecting This: Faithfulness Checking
# Conceptual — check whether claims in the answer are actually
# supported by the retrieved context
def check_faithfulness(answer, retrieved_context):
claims = extract_claims(answer)
for claim in claims:
if not is_supported_by(claim, retrieved_context):
flag_unfaithful_claim(claim)This is often done with a separate LLM call acting as a judge (see Faithfulness), checking each claim in the answer against the actual retrieved text.
Mitigation Strategies Specific to RAG
- Explicit prompt instructions to use ONLY the provided context (see RAG Prompt)
- An explicit, required fallback response when context is insufficient
- Faithfulness evaluation as an ongoing production monitoring signal, not just a pre-launch check
- Lower temperature for fact-sensitive RAG applications
- Requiring citations, which both encourages grounding and makes unsupported claims easier for a human to spot
Practical Use Case
A legal or medical RAG application needs faithfulness checking as a genuine production safeguard, not an optional nicety — the cost of an unfaithful, confidently-stated answer in these domains is high enough to justify the added evaluation overhead.
Common Mistakes
- Assuming "we use RAG" is sufficient protection against hallucination on its own, without prompt-level grounding instructions or faithfulness monitoring
- Not distinguishing "the model hallucinated despite good context" from "retrieval failed to find good context" when debugging a bad answer
Interview Relevance
"Does RAG completely solve hallucination?" — no; a strong answer explains that RAG reduces but doesn't eliminate the risk, and names at least one specific way a model can still hallucinate even with correct retrieved context.
Practice Question
Given a retrieved context stating "Standard shipping takes 5-7 business days" and a model answer stating "Your order will arrive in 5-7 business days via express shipping," identify the unfaithful claim.
