RAG Systems in Practice — From Prototype to Production

RAG Systems in Practice — From Prototype to Production

Retrieval-Augmented Generation (RAG) has quickly become the go-to pattern for grounding Large Language Models in domain-specific knowledge. The concept is straightforward: instead of fine-tuning a model on your data, you retrieve relevant documents at query time and include them in the prompt context. But the gap between a working prototype and a production-ready RAG system is enormous.

This post covers the practical decisions and pitfalls you will encounter when building RAG systems that need to work reliably, at scale, and within budget.

Why RAG?

LLMs have a knowledge cutoff and no access to your private data. Fine-tuning is expensive, slow, and requires retraining whenever your data changes. RAG solves this by decoupling the knowledge source from the model:

1. Index your documents by converting them to vector embeddings
2. Retrieve relevant chunks based on the user''s query
3. Generate a response using the retrieved context

This gives you up-to-date, source-grounded answers without retraining the model.

Chunking: The Foundation You Cannot Ignore

The single most impactful decision in a RAG pipeline is how you split your documents into chunks. Get this wrong, and no amount of sophisticated retrieval will save you.

Common chunking strategies:

• Fixed-size chunks (e.g., 512 tokens with 50-token overlap): Simple, predictable, but often splits semantic units in half
• Recursive text splitting: Splits by paragraphs, then sentences, then characters — respects natural boundaries better
• Semantic chunking: Uses embedding similarity to detect topic boundaries — highest quality, highest cost
• Document-structure-aware splitting: Uses headers, sections, and formatting cues from the source document

Here is a practical chunking implementation using LangChain:

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=100,
    separators=["\n\n", "\n", ". ", " ", ""],
    length_function=len,
)

chunks = splitter.split_documents(documents)

My recommendation: Start with recursive text splitting at 500–800 tokens with 10–15% overlap. Measure retrieval quality, then adjust. Smaller chunks generally improve precision but hurt context completeness.

Choosing an Embedding Model

Your embedding model converts text to dense vectors that capture semantic meaning. The choice matters more than most people realize.

Key considerations:

• Dimensionality: Higher dimensions capture more nuance but increase storage costs and retrieval latency
• Multilingual needs: If your content spans languages, test multilingual models explicitly
• Cost at scale: At millions of documents, embedding costs add up quickly — open-source models deployed on your infrastructure can be 10x cheaper

Vector Databases: Where Your Embeddings Live

You need a vector database to store embeddings and perform similarity search efficiently. Here are the main options:

Pinecone: Fully managed, excellent developer experience, pay-per-query pricing. Great for getting started, but vendor lock-in and costs can surprise you at scale.

Weaviate: Open-source, supports hybrid search (vector + keyword), built-in modules for popular embedding models. Good middle ground.

pgvector: PostgreSQL extension for vector similarity search. If you are already running PostgreSQL (e.g., via Supabase), this is compelling — no new infrastructure needed.

Qdrant: Open-source, high performance, rich filtering capabilities. My pick for self-hosted production deployments.

Here is a minimal example using pgvector with Supabase:

import openai
from supabase import create_client

supabase = create_client(SUPABASE_URL, SUPABASE_KEY)

# Generate embedding
response = openai.embeddings.create(
    model="text-embedding-3-small",
    input="How do I implement rate limiting?"
)
query_embedding = response.data[0].embedding

# Search for similar documents
result = supabase.rpc(
    "match_documents",
    {
        "query_embedding": query_embedding,
        "match_threshold": 0.78,
        "match_count": 5
    }
).execute()

for doc in result.data:
    print(f"Score: {doc['similarity']:.3f} — {doc['content'][:100]}")

Re-Ranking: The Quality Multiplier

Initial vector retrieval often returns results that are semantically adjacent but not precisely relevant. Re-ranking applies a more sophisticated model to the top-k results to improve ordering.

from cohere import Client

co = Client(api_key=COHERE_API_KEY)

# Re-rank the initial retrieval results
reranked = co.rerank(
    model="rerank-english-v3.0",
    query=user_query,
    documents=[doc["content"] for doc in initial_results],
    top_n=3
)

final_docs = [initial_results[r.index] for r in reranked.results]

Re-ranking typically improves answer quality by 15–25% in my experience, at the cost of an additional API call adding 100–200ms of latency. For production systems where answer quality matters, it is almost always worth it.

Common Pitfalls

1. Chunks too large. If your chunks are 2000+ tokens, you retrieve fewer of them (context window limits), and each chunk contains diluted information. The LLM struggles to find the relevant needle in the haystack.

2. No metadata filtering. Without metadata (document type, date, author, category), you cannot scope retrieval. A question about "Q4 2025 revenue" should not retrieve documents from 2023.

3. Ignoring retrieval quality metrics. You must measure how well your retrieval performs independently of generation. Key metrics:

• Recall@k: How many of the relevant documents are in the top-k results?
• MRR (Mean Reciprocal Rank): How high is the first relevant result ranked?
• Precision@k: What fraction of retrieved documents are actually relevant?

4. No evaluation dataset. Build a golden set of 50–100 question-answer pairs with source references. Test every pipeline change against this set. Without this, you are optimizing blind.

5. Neglecting latency and cost monitoring. In production, track: embedding latency, retrieval latency, re-ranking latency, total tokens sent to the LLM, and cost per query. Set alerts for anomalies.

Production Checklist

Before going live, ensure you have:

• Chunking strategy validated against retrieval quality metrics
• Embedding model benchmarked on your actual data
• Metadata schema defined and populated
• Re-ranking evaluated (cost vs. quality trade-off)
• Evaluation dataset with at least 50 question-answer pairs
• Latency budgets defined (e.g., p95 under 3 seconds)
• Cost monitoring and alerting in place
• Fallback behavior when retrieval returns no relevant results
• Content update pipeline (how new/updated documents get re-indexed)

Conclusion

Building a RAG prototype takes an afternoon. Building a production RAG system takes weeks of careful iteration on chunking, retrieval quality, and operational concerns. The key insight is that RAG is fundamentally an information retrieval problem with a generative layer on top — and most of your effort should go into getting the retrieval right.

Start simple, measure everything, and resist the temptation to add complexity before you understand where your pipeline is actually failing.