Reranking

Reranking adds a second, more precise relevance-scoring pass over an initial set of retrieved candidates — trading a bit of extra latency for meaningfully better final ranking than the initial vector search alone typically achieves.

Why a Second Pass Helps

Stage 1 — Initial retrieval (fast, broad):
  Vector search returns top 20 candidates, ranked by embedding
  similarity — fast, but embedding similarity is a somewhat
  coarse relevance signal.

Stage 2 — Reranking (slower, more precise):
  A dedicated reranking model examines the query and EACH of
  the 20 candidates together (not just comparing precomputed
  vectors), producing a more accurate relevance score.
  → Reorders the 20 candidates, often surfacing a truly more
    relevant result that ranked lower in stage 1.

Final: take the top 3-5 AFTER reranking, not after initial
       retrieval alone.

Why Not Just Rerank Everything From the Start?

Reranking models are typically more computationally expensive per comparison than vector similarity search — practical to run against a small candidate set (say, 20-50 initial results) but too slow to run against an entire large collection directly. This is why the two-stage pattern exists: fast, broad retrieval first, then a more expensive, precise reranking pass over just the promising candidates.

Conceptual Flow

candidates = vector_db.query(query_embedding, top_k=20)
reranked = reranker_model.score(query, candidates)
final_results = reranked[:5]  # the actual context sent to the LLM

Practical Use Case

Reranking is one of the highest-leverage additions to a RAG system once basic retrieval is working — it's a common, well-documented technique for meaningfully improving answer quality without needing to change the underlying embedding model or vector database.

Common Mistakes

• Adding reranking without measuring whether it actually improves results for your specific content — while it commonly helps, it's still worth verifying against your own evaluation set
• Reranking too small an initial candidate set (e.g. only 3-5), leaving little room for reranking to actually improve the final selection
• Not accounting for the added latency reranking introduces, especially for latency-sensitive real-time applications

Interview Relevance

"Why would you add a reranking step after vector retrieval instead of just retrieving more results directly?" — reranking uses a more precise (but more expensive) relevance signal than embedding similarity alone, applied selectively to a manageable candidate set rather than the whole collection.

Practice Question

Explain why reranking 20 initial candidates down to the best 5 typically produces better final results than retrieving only 5 directly from vector search.

Related Notes

• Vector Retrieval
• Hybrid RAG
• Advanced RAG