RAG architecture splits cleanly into two phases: an offline ingestion phase (done once, or whenever content changes) and an online query phase (done for every user question) — understanding this split clarifies what needs to be fast/cheap versus what can be done ahead of time.
Two Distinct Phases
OFFLINE — INGESTION (runs once per document, or on content updates)
Document → extract text → chunk → embed → store in vector DB
ONLINE — QUERY (runs on every single user question)
User question → embed → search vector DB → retrieve top-k chunks
→ build prompt with retrieved context → LLM generates answerThe expensive, batch-friendly work (processing potentially thousands of documents) happens offline, ahead of time. The query-time path only needs to embed one short question and search an already-built index — this is why RAG can feel fast to users even though document processing itself takes real time.
Components Involved
| Component | Role |
|---|---|
| Document loader / parser | Extracts raw text from source documents (PDFs, web pages, etc.) — see Document Processing |
| Chunker | Splits text into retrieval-sized pieces — see Document Chunking |
| Embedding model | Converts chunks (and queries) into vectors — see Embedding Models |
| Vector database | Stores vectors and serves similarity search — see Vector Database |
| Retriever | The logic that queries the vector DB and returns relevant chunks — see Vector Retrieval |
| LLM | Generates the final answer using retrieved context — see RAG Prompt |
Practical Use Case
Understanding the offline/online split matters for real system design: content update frequency (how often ingestion needs to re-run), and query latency budget (what has to happen fast, within a user's request) are two separate engineering concerns that this architecture split makes explicit.
Common Mistakes
- Re-processing (parsing, chunking, embedding) documents on every query instead of once during ingestion — a serious, avoidable latency and cost problem
- Not planning for how content updates propagate — if a source document changes, stale chunks need to be identified and re-ingested, not just silently left outdated
Interview Relevance
"Why doesn't RAG re-process the entire document collection on every user query?" — because ingestion (parsing, chunking, embedding, storing) happens offline, ahead of time; only the query itself is embedded and searched at request time.
Practice Question
Sketch which parts of a RAG system need to run in real time (within a user's request) versus which can run as a background/batch process.
