RAG Is Not Just Vector Search: Building Production Retrieval Systems

The RAG Hype Cycle

Retrieval-Augmented Generation exploded onto the scene as the answer to LLM hallucination. The idea is seductively simple: instead of relying on the model's parametric knowledge, retrieve relevant documents and include them in the prompt.

The basic pipeline looks like this:

1. User asks a question
2. Embed the question using an embedding model
3. Search a vector database for similar document chunks
4. Stuff the top-k results into the prompt
5. Let the LLM generate an answer

This works impressively well in demos. It fails impressively badly in production.

Why Naive RAG Fails

The Chunking Problem

Most tutorials chunk documents into fixed-size pieces. A 500-token chunk might split a paragraph mid-sentence, separate a code block from its explanation, or cut a table in half.

The result: retrieved chunks contain partial information that confuses the LLM rather than helping it.

Better approaches:

• Semantic chunking: Split on paragraph or section boundaries
• Hierarchical chunking: Store both large and small chunks, retrieve at multiple granularities
• Overlapping windows: Include context from adjacent chunks

def semantic_chunk(text: str, max_tokens: int = 500) -> list[str]:
    paragraphs = text.split("\n\n")
    chunks = []
    current_chunk = []
    current_length = 0
 
    for paragraph in paragraphs:
        para_tokens = count_tokens(paragraph)
        if current_length + para_tokens > max_tokens and current_chunk:
            chunks.append("\n\n".join(current_chunk))
            current_chunk = []
            current_length = 0
        current_chunk.append(paragraph)
        current_length += para_tokens
 
    if current_chunk:
        chunks.append("\n\n".join(current_chunk))
    return chunks

The Relevance Problem

Vector similarity is not the same as relevance. A query about "Python error handling best practices" might retrieve chunks about Python error messages, Python exception classes, or JavaScript error handling — all semantically similar but not all relevant.

Solutions:

• Hybrid search: Combine vector similarity with keyword search (BM25). Vector search captures semantic meaning; keyword search captures exact terminology
• Re-ranking: Use a cross-encoder model to re-score retrieved candidates. This is more expensive but dramatically more accurate
• Metadata filtering: Filter by document type, date, or source before vector search

The Context Window Problem

Stuffing all retrieved chunks into the prompt creates noise. If 3 of 10 retrieved chunks are relevant, the 7 irrelevant ones can distract the LLM and degrade answer quality.

Solutions:

• Aggressive re-ranking: Only keep top 3-5 truly relevant chunks
• Compression: Use an LLM to summarize retrieved chunks before including them
• Citation tracking: Ask the model to cite which chunks it used, then verify

A Production RAG Architecture

Here is what a battle-tested RAG system looks like:

Query → Query Expansion → Hybrid Retrieval → Re-ranking → Context Assembly → LLM → Answer Validation

Query Expansion

The user's query is often too short or ambiguous. Generate 2-3 alternative phrasings:

async def expand_query(query: str) -> list[str]:
    prompt = f"""Generate 3 alternative search queries for: "{query}"
    Return only the queries, one per line."""
    alternatives = await llm.generate(prompt)
    return [query] + alternatives.strip().split("\n")

Search with all expanded queries and merge results, giving higher weight to chunks that appear across multiple queries.

Hybrid Retrieval

Run both vector search and keyword search in parallel:

async def hybrid_retrieve(query: str, k: int = 20):
    vector_results, keyword_results = await asyncio.gather(
        vector_store.search(query, k=k),
        keyword_store.search(query, k=k),
    )
    return reciprocal_rank_fusion(vector_results, keyword_results)

Reciprocal Rank Fusion (RRF) is a simple and effective way to merge ranked lists from different search methods.

Re-Ranking

Use a cross-encoder model to re-score the merged results:

from sentence_transformers import CrossEncoder
 
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
 
def rerank(query: str, documents: list[str], top_k: int = 5):
    pairs = [[query, doc] for doc in documents]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(scores, documents), reverse=True)
    return [doc for _, doc in ranked[:top_k]]

Answer Validation

After the LLM generates an answer, validate it:

1. Groundedness check: Can every claim in the answer be traced to a retrieved chunk?
2. Relevance check: Does the answer actually address the user's question?
3. Confidence score: If the model is uncertain, say so rather than hallucinate

Measuring RAG Quality

You need three metrics:

1. Retrieval precision: What fraction of retrieved chunks are actually relevant?
2. Retrieval recall: What fraction of relevant chunks were retrieved?
3. Answer accuracy: Does the final answer correctly reflect the source documents?

Build an evaluation dataset of question-answer pairs with known source documents. Run this after every pipeline change.

Key Lessons

1. Chunking strategy matters more than embedding model choice
2. Hybrid search outperforms pure vector search in nearly every benchmark
3. Re-ranking is the highest-ROI improvement you can make
4. Query expansion catches the queries that vector search misses
5. Measure everything — without metrics, you are flying blind
6. Start simple, iterate based on failure analysis — log every query, retrieved context, and generated answer

RAG is not a single technique. It is a pipeline, and every stage can be optimized. The teams that succeed in production are the ones that instrument every step and iterate relentlessly on the failure cases.