Vector retrieval is the RAG-specific application of vector search (see Vector Search) — embedding a user's query and fetching the most relevant stored chunks to use as context for generation.
The Retrieval Step in Context
User question: "What's your policy on damaged items?"
↓ embed the question
Query vector
↓ search the vector database (built during ingestion)
Top-k most similar chunks, e.g.:
1. "Damaged items can be returned within 14 days..." (score: 0.89)
2. "Our packaging is designed to prevent damage..." (score: 0.71)
3. "General return policy overview..." (score: 0.65)
↓
These chunks become the context inserted into the RAG promptChoosing top_k — How Many Chunks to Retrieve
| top_k Value | Tradeoff |
|---|---|
| Too low (e.g. 1-2) | Risk of missing genuinely relevant content that didn't rank at the very top |
| Too high (e.g. 20+) | More irrelevant/noisy content in the prompt, consuming context budget and potentially diluting focus |
| Common starting range | Often 3-10, tuned per use case — not a universal number |
Retrieval Quality Is Not Guaranteed by "It Returned Something"
A vector search always returns its top-k closest matches — even if none of them are actually good matches for the query. A similarity threshold (rejecting matches below a certain score) or explicit "no relevant content found" handling is worth adding, rather than always feeding whatever was retrieved into the prompt regardless of actual relevance.
Practical Use Case
This retrieval step is the single highest-leverage point for improving RAG answer quality in most systems — if the wrong chunks are retrieved, no amount of clever prompt engineering downstream can produce a correct, grounded answer.
Common Mistakes
- Always using a fixed top_k regardless of similarity scores, including low-relevance matches when nothing genuinely relevant exists
- Not surfacing retrieval quality (similarity scores) in logs/monitoring, making it hard to diagnose whether a bad answer stemmed from bad retrieval or bad generation
Interview Relevance
"How would you handle a RAG query where none of the retrieved chunks are actually relevant?" — a similarity threshold and an explicit "insufficient information" response path, rather than blindly using whatever top-k results came back, is the expected engineering answer.
Practice Question
Design a similarity-threshold check that causes the system to respond "I don't have information about that" instead of generating an answer from weakly-relevant retrieved chunks.
