"Context" in a prompt is any information supplied to help the model answer accurately — retrieved documents, prior conversation, structured data — as opposed to relying purely on what the model memorized during training.
Why Providing Context Beats Relying on Memorized Knowledge
Without context:
"What's our refund policy?"
→ model has no way to know your company's specific policy;
it may guess, decline, or (worse) confidently hallucinate
a plausible-sounding but wrong answer
With context:
"Using only the policy text below, answer: what's our refund policy?
Policy: Refunds are accepted within 30 days of purchase with
original receipt. Sale items are final sale.
Question: What's our refund policy?"
→ grounded, accurate answer, traceable to real source textThis is the core mechanism behind RAG — retrieving relevant context and inserting it into the prompt before generation.
Instructing the Model to Actually Use the Context
Simply including context doesn't guarantee the model prioritizes it over its own memorized knowledge — being explicit helps:
Weak: "Here's some info: [context]. Answer: [question]"
Better: "Answer the question using ONLY the information in the
context below. If the answer isn't in the context, say
'I don't have that information' rather than guessing.
Context: [context]
Question: [question]"Context Competes for Space
Every piece of context added consumes part of the context window and adds to input token cost — more context isn't automatically better; irrelevant context can dilute focus and increase cost without improving (and sometimes hurting) answer quality.
Practical Use Case
This is the fundamental building block every RAG, document-QA, and grounded-chatbot application is built on — retrieve relevant context, insert it clearly into the prompt, and instruct the model to rely on it specifically rather than its own general knowledge.
Common Mistakes
- Including context without explicitly instructing the model to prioritize it — the model may still blend in its own memorized (possibly outdated or wrong) knowledge
- Dumping large amounts of loosely relevant context "just in case" instead of retrieving focused, genuinely relevant material
- Not giving the model a way to say "not found in context" — without this, it may fabricate an answer rather than admit the context doesn't cover the question
Interview Relevance
"Why might a RAG system still give a wrong answer even when the correct information was in the retrieved context?" — often, the prompt didn't clearly instruct the model to prioritize the provided context over its own memorized knowledge, or didn't offer a graceful "not found" option.
Practice Question
Write a prompt template that includes retrieved context and explicitly handles the case where the context doesn't contain the answer.
