How to Build a RAG System (2026 Guide)

Everyone is talking about RAG. Most of them are doing it wrong.

Retrieval Augmented Generation is the most practical AI architecture shipping right now. (For the protocol layer connecting these systems to agents, see our guide on Model Context Protocol.) It's how you make an LLM actually useful with your own data — without fine-tuning, without training from scratch, without spending six figures on GPU time.

We run a RAG system in production at Like One. Over 2,400 vectors across 7 collections, powering our AI brain with instant recall across hundreds of documents. This guide is what we learned building it.

What RAG Actually Is (30-Second Version)

An LLM knows what it was trained on. It doesn't know your company docs, your meeting notes, your product specs, or anything that happened after its training cutoff.

RAG fixes this in three steps:

1. Chunk your documents into pieces
2. Embed those chunks into vectors (numerical representations of meaning)
3. Retrieve the most relevant chunks when a user asks a question, and feed them to the LLM as context

That's it. No mysticism. You're giving the model a cheat sheet before it answers.

The Architecture That Actually Works

After building multiple RAG systems, here's the stack we recommend:

Embedding Model: Use a dedicated embedding model, not your chat model. OpenAI's text-embedding-3-small or Cohere's embed-v4 are both excellent. Local options like nomic-embed-text via Ollama work great if you want zero API costs.

Vector Store: ChromaDB for prototyping and small-to-medium scale. It's local, fast, and requires zero infrastructure. Pinecone or Weaviate if you need managed hosting at scale. We run ChromaDB in production with 2,400+ vectors and it handles sub-second queries without breaking a sweat.

Chunking Strategy: This is where most people mess up. More on this below.

LLM: Claude (our pick), GPT-4, or Gemini for the generation step. The model matters less than your retrieval quality. Set up custom instructions to get the best generation quality.

Step 1: Chunking — The Make-or-Break Decision

Bad chunking is the #1 reason RAG systems give bad answers. Here's what actually works:

Chunk Size

Aim for 200-500 tokens per chunk. Smaller chunks give more precise retrieval. Larger chunks give more context. The sweet spot depends on your content:

• Technical docs: 200-300 tokens. You want precise function-level retrieval.
• Business content: 300-500 tokens. Broader context helps with nuanced questions.
• Legal/compliance: 400-600 tokens. You need full clause context.

Overlap

Always use overlapping chunks — 10-15% overlap prevents information from getting split across chunk boundaries. If your chunk size is 400 tokens, use a 50-token overlap.

Semantic Chunking vs. Fixed-Size

Fixed-size chunking (split every N tokens) is fast and predictable. Semantic chunking (split at natural boundaries like paragraphs or sections) preserves meaning better but is harder to implement consistently.

Our recommendation: start with fixed-size + overlap. Move to semantic chunking only when you have evidence that chunk boundary issues are hurting retrieval quality.

Step 2: Embeddings — Getting Them Right

Embeddings convert text into vectors — arrays of numbers that capture semantic meaning. Similar concepts end up near each other in vector space.

# Python example with ChromaDB
import chromadb

client = chromadb.PersistentClient(path="./my_rag_db")
collection = client.get_or_create_collection(
    name="knowledge_base",
    metadata={"hf:space": "default"}
)

# Add documents
collection.add(
    documents=["Your document text here", "Another document"],
    ids=["doc1", "doc2"],
    metadatas=[{"source": "manual"}, {"source": "auto"}]
)

# Query
results = collection.query(
    query_texts=["How does the billing system work?"],
    n_results=5
)

Key principles:

• Use the same embedding model for indexing and querying. Mixing models produces garbage results.
• Store metadata with every chunk. Source file, date, section title, document type. You'll need this for filtering and attribution.
• Normalize your text before embedding. Strip excessive whitespace, fix encoding issues, remove boilerplate headers/footers. Garbage in, garbage out.

Step 3: Retrieval — Quality Over Quantity

Retrieval is where your system earns its keep. The goal: find the 3-5 most relevant chunks for any given query.

Basic Similarity Search

Start here. Cosine similarity between the query embedding and your stored embeddings. Every vector database supports this out of the box.

Hybrid Search (The Upgrade)

Combine vector similarity with keyword search (BM25). Vector search catches semantic matches ("revenue" finds "sales figures"). Keyword search catches exact matches ("Q3-2026" finds that exact string). Together, they cover each other's blind spots.

Reranking (The Secret Weapon)

After your initial retrieval, run the results through a reranker model. Cohere Rerank or a cross-encoder model re-scores your results with much higher accuracy than embedding similarity alone. This single step can improve answer quality by 20-30%.

# Pseudo-code for a hybrid + rerank pipeline
initial_results = vector_db.query(question, n=20)  # Cast a wide net
keyword_results = bm25_search(question, n=20)       # Keyword backup
merged = deduplicate(initial_results + keyword_results)
final = reranker.rank(question, merged, top_n=5)    # Precision pass

Step 4: Generation — Prompting With Retrieved Context

Once you have your relevant chunks, feed them to the LLM with a structured prompt:

You are a helpful assistant. Answer the user's question 
using ONLY the provided context. If the context doesn't 
contain enough information, say so — don't make things up.

Context:
{retrieved_chunks}

Question: {user_question}

Critical rules:

• Always include a "don't hallucinate" instruction. LLMs will fill gaps with plausible-sounding fiction if you let them.
• Cite your sources. Include chunk metadata in your prompt and ask the model to reference which documents it used.
• Set a token budget. Don't dump 50 chunks into context. 3-5 high-quality chunks beat 20 mediocre ones every time.

Production Lessons (What We Learned the Hard Way)

1. Freshness Matters

Your knowledge base is only as good as its last update. We run automated embedding pipelines every 5 minutes. If your data changes daily, batch-update nightly at minimum. Stale vectors give stale answers.

2. Collections Are Your Friend

Don't dump everything into one giant collection. Separate by domain: one collection for docs, another for meeting notes, another for code. Query the relevant collection(s) based on the question type. We use 7 collections — each one tuned for its content type.

3. Evaluation Is Non-Negotiable

Build an eval set: 50-100 question-answer pairs with known correct answers. Run them through your pipeline weekly. Track:

• Retrieval precision: Are the right chunks being found?
• Answer accuracy: Is the generated answer correct?
• Hallucination rate: How often does the model invent information?

If you don't measure, you don't know if your changes help or hurt.

4. Handle "I Don't Know" Gracefully

The best RAG systems know when they don't have the answer. If retrieval confidence is low (similarity scores below your threshold), return "I don't have enough information to answer that" instead of a confident-sounding wrong answer. Users trust systems that admit limitations far more than systems that bluff.

5. Chunk Your Code Differently Than Your Prose

Code needs different chunking than text. Use AST-aware chunking for code (split at function/class boundaries). Embedding a random 300-token slice of a Python file produces awful results. Embedding a complete function with its docstring produces great ones.

Advanced RAG Patterns: Beyond Basic Retrieval

Basic RAG — embed, retrieve, generate — gets you 70% of the way there. The remaining 30% requires patterns that most tutorials skip.

Query expansion. Users ask bad questions. Not because they're bad at asking — because they don't know the vocabulary of your knowledge base. "How do I fix the login thing?" should retrieve documents about authentication errors, session management, and OAuth token refresh. Query expansion rewrites the user's question into multiple queries that cover different phrasings and related concepts. You can use the LLM itself for this — ask it to generate three alternate phrasings before retrieval.

Parent-child chunking. Store small chunks (200 tokens) for precise retrieval, but return the parent chunk (800 tokens) for context. This gives you the precision of small chunks with the context of large ones. When a 200-token chunk matches, you pull its parent section — the full paragraph or subsection it came from — and feed that to the LLM. Best of both worlds.

Multi-index routing. Different question types need different retrieval strategies. A factual question ("what's our refund policy?") needs keyword-heavy retrieval. A conceptual question ("how does our pricing compare to competitors?") needs vector-heavy retrieval. A temporal question ("what changed in the last release?") needs freshness-weighted retrieval. Build a lightweight classifier that routes queries to the right retrieval strategy.

Contextual compression. After retrieval, you often have more context than the LLM needs. Contextual compression uses a smaller, faster model to extract only the relevant sentences from each retrieved chunk before passing them to the main LLM. This reduces token usage (and cost) while improving answer quality by removing noise. (This is the same principle behind the boot context approach in persistent memory systems.)

RAG in Production: Monitoring and Maintenance

Shipping a RAG system is maybe 40% of the work. Keeping it healthy in production is the other 60%.

Embedding drift. When you update your embedding model (and you will — newer models are consistently better), every vector in your store becomes incompatible. You can't mix vectors from different models in the same collection. Plan for full re-embedding from day one. Keep your source documents in a format you can re-process, version your embedding pipeline, and schedule re-embedding as a migration step when you upgrade models.

Retrieval monitoring. Log every query, the chunks retrieved, and whether the user found the answer helpful (thumbs up/down, follow-up questions, or session abandonment as a proxy). Build a dashboard that surfaces low-retrieval-confidence queries — these are the gaps in your knowledge base. A self-improving agent can even flag these gaps automatically and suggest documents to add.

Index freshness. We run automated embedding pipelines every 5 minutes, but even that can miss structural changes. When a document is deleted or substantially rewritten, old vectors linger as ghosts — they match queries but point to content that no longer exists. Implement a reconciliation step that compares your source documents against your vector store and purges orphaned vectors. Run it daily.

Cost tracking. RAG costs come from three places: embedding API calls, vector storage, and generation API calls. Embedding is usually cheap (pennies per thousand chunks). Generation is where costs spike — especially if you're feeding 5 chunks of 500 tokens each into every query. Track cost per query and set alerts. If a single query costs more than your threshold, investigate whether retrieval is returning too many chunks or the wrong ones.

Latency budgets. Users expect answers in 2-3 seconds. Your RAG pipeline has to fit embedding the query (~100ms), vector search (~50ms), optional reranking (~200ms), and LLM generation (~1-2s) into that window. Profile each stage and optimize the bottleneck. In our experience, reranking and generation dominate latency — retrieval itself is rarely the problem.

One underappreciated pattern: feedback loops. When users mark an answer as unhelpful, log the query and the retrieved chunks. Review these weekly. They reveal exactly where your chunking, embedding, or retrieval strategy is failing — and they're more valuable than any benchmark dataset because they reflect real usage patterns from your actual users.

Common Mistakes to Avoid

Production RAG Systems

Building RAG for your organization? Our consulting team designs retrieval systems that actually work — with proper chunking, embedding, and reranking architectures.

• Chunking too large. 2,000-token chunks feel safe but dilute retrieval precision. You'll get the right document but the wrong paragraph.
• Skipping metadata. Without source tracking, your system can't cite its sources and you can't debug retrieval failures.
• Ignoring the embedding model. Not all embedding models are equal. Test at least two on your actual data before committing.
• Over-engineering on day one. Start with basic similarity search. Add hybrid search when you have evidence it's needed. Add reranking when you have evidence hybrid isn't enough. Premature optimization kills RAG projects.
• Not versioning your index. When you change chunking strategy or embedding model, you need to re-embed everything. Track which version of your pipeline produced each vector.

The Bottom Line

RAG is the bridge between "AI that sounds smart" and "AI that knows your stuff." The architecture is straightforward: chunk, embed, retrieve, generate. The execution is where it gets interesting.

Start simple. Measure everything. Iterate on the parts that actually matter — chunking strategy and retrieval quality — before reaching for exotic techniques.

The companies winning with AI right now aren't the ones with the fanciest models. They're the ones with the best retrieval pipelines feeding those models exactly the right context at exactly the right time.

---

Like One Academy has a full course on building RAG systems — from first embed to production deployment. Our nonprofit academy covers AI engineering across 52 courses with structured learning paths. Start learning free.