Retrieval-Augmented Generation has moved from research novelty to the default architecture for knowledge-intensive LLM applications. But the gap between a working RAG demo and a production RAG system is enormous. After shipping RAG systems across healthcare, logistics, fintech, and legal domains, here is what we have learned about building ones that actually work at scale.
Why most RAG implementations fail
The typical RAG failure mode is not technical — it is architectural. Teams build a pipeline that works well on their test queries, deploy it, and discover that real user queries are messier, more ambiguous, and structurally different from the cases they tested.
The root causes cluster around three areas: retrieval quality (the wrong chunks are being returned), generation quality (the LLM is not using retrieved context well), and evaluation (there is no systematic way to measure whether the system is actually getting better or worse).
Each of these is solvable, but they require deliberate design decisions — not just plugging a vector database in front of an LLM.
Chunking strategy matters more than most engineers realise
The default approach of splitting documents into fixed-size chunks by token count is almost always the wrong choice for production. Fixed-size chunking ignores document structure — it splits sentences mid-thought, separates headings from their content, and creates chunks that are contextually meaningless.
For structured documents (contracts, reports, technical documentation), use hierarchical chunking: preserve the document structure and index at multiple granularities (section, subsection, paragraph). For unstructured text, use semantic chunking — split at natural semantic boundaries rather than token counts.
One pattern that consistently outperforms fixed-size chunking is the parent-child chunk approach: index small, precise child chunks for retrieval, but return the larger parent chunk to the LLM as context. This gives you retrieval precision without losing contextual completeness.
Hybrid retrieval is not optional at production scale
Pure vector search (dense retrieval) excels at semantic similarity but struggles with exact keyword matching, proper nouns, and queries that include specific codes, identifiers, or technical terms. Pure BM25 (sparse retrieval) handles keywords well but misses semantic relationships.
In production, you need both. Reciprocal Rank Fusion (RRF) is the simplest effective fusion strategy — it combines rankings from dense and sparse retrieval without requiring a learned fusion model. For most domains, dense:sparse weighting in the range of 70:30 to 80:20 is a reasonable starting point.
Beyond the retrieval algorithm, invest in your embedding model selection. General-purpose embeddings work acceptably, but domain-specific embeddings — or general models fine-tuned on your document corpus — consistently deliver 15 to 25 percent retrieval accuracy improvement.
Query transformation is where you earn your retrieval gains
User queries are rarely optimal for retrieval. They are short, ambiguous, implicitly referential, or phrased in ways that do not match how your source documents are written.
Query expansion using an LLM to generate multiple retrieval queries from a single user question is one of the highest-leverage interventions you can make. HyDE (Hypothetical Document Embeddings) — generating a hypothetical answer and using that as the retrieval query — works particularly well for technical domains where question and answer phrasing differ significantly.
For multi-turn conversations, build a query rewriting step that resolves coreferences and incorporates conversation history before hitting retrieval. This is consistently one of the top sources of production quality improvement.
Evaluation infrastructure should be built before the pipeline
This is the most frequently skipped step, and the one that hurts the most. Without a systematic evaluation framework, you are guessing whether changes to your pipeline improve or degrade quality.
At minimum, build an offline evaluation suite covering retrieval quality (are the right chunks being returned?), faithfulness (is the answer grounded in the retrieved context?), and answer quality (is the answer correct and complete?). Tools like RAGAS provide good starting point metrics for all three dimensions.
Beyond offline evaluation, instrument your production pipeline to log retrieval inputs and outputs, flag low-confidence responses, and capture implicit feedback signals. The production logs from real user queries are your most valuable dataset for continued improvement.
Latency optimisation for production workloads
A RAG pipeline adds retrieval, reranking, and sometimes query transformation steps to the base LLM latency. Without optimisation, this can push end-to-end latency to unacceptable levels for interactive applications.
The highest-leverage latency optimisations are: (1) async retrieval — run dense and sparse retrieval in parallel; (2) result caching — cache retrieval results for common or repeated queries; (3) reranker placement — run reranking only when the initial retrieval pool is large, and skip it for high-confidence single-document queries; (4) streaming — stream LLM responses to the user before the full generation is complete.
For most applications, well-implemented retrieval caching alone reduces p95 latency by 40 to 60 percent for repeated or similar queries.