How to Build a RAG System from Scratch: A Complete Guide

Large Language Models (LLMs) like ChatGPT are impressive, but they have a major flaw: they can hallucinate. They make things up because they only know what they were trained on. If you ask about your company's internal policies or a specific document, they will either guess incorrectly or say they don't know.

RAG solves this problem. By building a RAG system, you give your AI the ability to "look up" answers in your own documents before responding. Instead of guessing, it retrieves relevant information and uses that as context to generate a factual, grounded answer .

In this guide, I'll walk you through building a complete RAG system from scratch using Python, a vector database, and an LLM API. You'll learn how RAG works under the hood without relying on complex frameworks, and you'll have a working system by the end.

What Is RAG and Why Do You Need It?

Retrieval-Augmented Generation (RAG) is a technique that combines information retrieval with large language models. Here is how it works in simple terms:

1. You upload documents into a database.

2. When a user asks a question, the system searches the database for the most relevant pieces of information .

3. It then sends that information to the LLM along with the question.

4. The LLM generates an answer based only on the provided context .

Why this matters: Instead of asking a model to answer from memory (where it might hallucinate), you help it "look things up" before it speaks . This drastically reduces hallucinations, allows you to use your own private data without retraining the model, and ensures answers are grounded in your actual documents .

The Architecture: How RAG Works Under the Hood

A typical RAG pipeline follows these steps :

1. Document Ingestion (Indexing Phase)

• Load your documents (PDFs, text files, etc.).

• Split them into smaller chunks (because LLMs have token limits).

• Convert each chunk into a vector embedding (a list of numbers that captures meaning).

• Store the chunks and their embeddings in a vector database.

2. Query Processing (Retrieval Phase)

• A user types a question.

• Convert the question into an embedding using the same model.

• Perform a similarity search in the vector database to find chunks that are mathematically close to the question embedding .

• Retrieve the top-k most relevant chunks.

3. Generation (Answer Phase)

• Build a prompt that includes: a system instruction, the retrieved context, and the user's question.

• Send this augmented prompt to an LLM.

• The LLM generates a response based only on the provided context .

Step-by-Step: Building Your RAG System

We'll build a simple yet complete RAG system that answers questions from a PDF document. To make this free and easy to follow, we'll use Python, LangChain (for orchestration), ChromaDB (a local vector database), and Google Gemini (which has a generous free tier) .

What You Will Need

• Python 3.9+

• A Google Gemini API key (free from Google AI Studio)

• A sample PDF document to test with

Step 1: Set Up the Project

First, create a new project directory and set up a virtual environment:

mkdir my-rag-project
cd my-rag-project

# Create and activate a virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Install the required packages:

pip install langchain langchain-google-genai langchain-community chromadb python-dotenv pypdf

Create a .env file in your project root and add your Gemini API key:

GOOGLE_API_KEY=your_api_key_here

Step 2: Prepare Your Documents

Place a sample PDF in your project folder. The PyPDFLoader will handle extracting text from it.

Step 3: Write the RAG Code

Create a Python file rag_app.py and build the pipeline:

Load and Split Documents

The first step is loading your PDF and splitting it into chunks. Chunking is critical because LLMs have token limits, and smaller chunks make retrieval more precise . The RecursiveCharacterTextSplitter intelligently splits by paragraphs and sentences to preserve meaning .

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

loader = PyPDFLoader("your_sample.pdf")
documents = loader.load()

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,        # Target tokens per chunk
    chunk_overlap=200       # Overlap to preserve context at boundaries
)
chunks = splitter.split_documents(documents)

Why chunk size matters: The optimal chunk size depends on your content. Highly structured documents work better with smaller chunks. Overlapping chunks ensure that concepts spanning chunk boundaries appear in at least one complete chunk .

Create Embeddings and Vector Store

Next, convert the chunks into vector embeddings using Google's embedding model. Embeddings capture the semantic meaning of text as numbers, enabling similarity search . ChromaDB will store these vectors persistently .

from langchain_google_genai import GoogleGenerativeAIEmbeddings
from langchain_community.vectorstores import Chroma

embeddings = GoogleGenerativeAIEmbeddings(model="models/embedding-001")
vector_store = Chroma.from_documents(chunks, embeddings)

Build the RAG Pipeline

Now, create the retrieval and generation pipeline. The retriever finds the most relevant chunks. The LLM then uses those chunks as context to generate a grounded answer .

from langchain_google_genai import ChatGoogleGenerativeAI
from langchain.prompts import PromptTemplate
from langchain.chains import RetrievalQA

retriever = vector_store.as_retriever(search_kwargs={"k": 3})  # Retrieve top 3 chunks

llm = ChatGoogleGenerativeAI(model="gemini-1.5-flash", temperature=0.2)

prompt = PromptTemplate(
    template="""You are a helpful assistant. Use ONLY the following context to answer the question.
    Context: {context}
    Question: {question}
    Answer:""",
    input_variables=["context", "question"]
)

rag_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=retriever,
    return_source_documents=True
)

Key detail: The prompt instructs the LLM to use only the provided context. This prevents hallucinations. The temperature setting of 0.2 keeps responses deterministic and grounded .

Query the System

Finally, test your RAG system with a question about your document.

result = rag_chain("What is the main topic of this document?")
print(result["result"])

Advanced Patterns: Beyond the Basics

Once you have a working RAG system, consider these production-grade improvements:

Context-Window Retrieval

Retrieve adjacent chunks along with the most relevant match. If chunk 47 matches the query, also fetch chunks 46 and 48 to preserve context. This reduces the risk of incomplete answers when concepts span chunk boundaries .

Query Translation

Transform the user's question before retrieval to improve results:

• Multi-Query: Generate multiple variations of the question to capture different angles.

• HyDE: Generate a hypothetical answer first, then use that to retrieve relevant documents.

• Step-back: Broaden the question to retrieve higher-level context .

Reranking and Metadata Filtering

Use a reranker model to reorder retrieved chunks based on relevance. Apply metadata filters (date, author, document type) to narrow the search space before retrieval .

Token Budget Management

LLMs have context windows. Track cumulative token counts during retrieval to ensure you stay within limits. Retrieve enough chunks to provide context but leave room for the system prompt and generated response .

Common Mistakes to Avoid

1. Skipping chunking optimization: Chunk size directly affects retrieval quality. Test different strategies on your data.

2. Ignoring retrieval evaluation: Without evaluating your retriever, you are flying blind. Use frameworks like RAGAS to measure performance.

3. Hardcoding secrets: Always use environment variables for API keys.

4. Not handling "I don't know": Your system should be able to say it does not know when no relevant chunks are retrieved.

The Bottom Line

Building a RAG system from scratch is the best way to understand how it works under the hood . While frameworks like LangChain simplify the process, knowing the fundamentals makes you a better engineer and helps you debug issues when they arise.

Your action plan:

• Today: Set up your environment and install dependencies

• This week: Build the basic RAG pipeline from this guide

• Next week: Experiment with different chunk sizes and retrievers

• Within a month: Add query translation, reranking, and evaluation

RAG is the backbone of modern AI applications. Master it, and you will be building production-grade systems faster than you think.

Build Your AI Career with Coding Now – Gurukul of AI

Whether you choose Data Scientist or AI Engineer, structured learning and hands-on practice are essential. At Coding Now – Gurukul of AI, we offer industry-oriented programs designed to take you from beginner to job-ready AI professional.

Our curriculum covers everything from Python fundamentals and statistical analysis to advanced topics like LLM APIs, RAG, orchestration frameworks, multi-agent systems, and production deployment. You will build practical, real-world projects under the guidance of experienced trainers and receive comprehensive career support.

With AI/ML hiring leading India's tech demand and 71% of employers prioritizing skills over degrees, there has never been a better time to invest in your AI career .

Visit us: https://codingnowai.in/ .

Contact Us

Phone: +91 9667708830
Email: info@codingnow.in
Website: https://codingnowai.in/

Address:
2nd Floor, Kapil Vihar (Opp. Metro Pillar No.354)
Pitampura, New Delhi – 110034

Backlink to main website: Explore Python and AI courses at Coding Now – Gurukul of AI