The Retrieval Is the Bug: What Breaks in Production RAG

A retrieval-augmented generation (RAG) demo always looks good. You load a handful of PDFs, split them into fixed-size chunks, embed them, retrieve the top five matches, and the model answers your test questions with clean citations. Then you ship it to real users and it quietly gets worse. Queries get ambiguous. The corpus grows. Answers start slipping, and there is no metric telling you why until users complain. Most teams blame the model and burn weeks swapping models and rewriting prompts. The 2026 data points somewhere else: roughly 80% of RAG failures trace back to the ingestion and chunking layer, not the LLM¹.

This article is the practical version of that finding. We walk the retrieval stack from chunking through evaluation, name the failure modes with the numbers behind them, and give you the levers that actually move production quality.

The model reasons over what retrieval hands it

The reason the model is the wrong place to look is structural. A generative model does not reason over your whole knowledge base. It reasons over the four to ten chunks that approximate-nearest-neighbor search happens to return². That selection defines the truth surface of the system. If the answer-bearing passage never reaches that window, no system prompt, no fine-tune, and no larger model can recover it. That is why retrieval is the single highest-leverage component in any RAG pipeline².

In enterprise settings, "RAG is broken" maps to three repeatable patterns³. Recall failure: the answer exists in the corpus, but retrieval never surfaces the passage that holds it. Ranking failure: retrieval returns candidates, but the best evidence is buried below loosely related passages. Grounding failure: the system answers confidently on thin, mismatched, or outdated evidence. All three happen before generation. All three are retrieval problems, and each needs a different fix.

The pipeline itself hides them. The standard flow embeds the user query, retrieves the top-k documents, and hands them to the model, and every arrow in that flow is a potential failure point⁴. Naive RAG masks the failure because it produces an answer regardless of whether retrieval was wrong⁴. So the first discipline is to stop treating generation as the stage to debug and start treating retrieval as a first-class engineering concern.

Chunking is the highest-leverage dial

Chunking defines the units of knowledge your system can retrieve, and it is the first thing to fix³. A chunk that is too large becomes topic soup: it looks relevant but does not answer the question cleanly. A chunk that is too small loses the definitions and dependencies that make a passage interpretable³. Bad chunking guarantees failure even with strong embeddings, hybrid search, and an expensive reranker³.

The strongest 2026 numbers come from strategy comparisons. A 2025 clinical decision-support study found adaptive, meaning-aware chunking hit 87% accuracy versus 13% for fixed-size baselines on the same corpus¹. NVIDIA's internal testing on university presentation decks found that hierarchical, parent-child chunking improved answer accuracy from 61% with fixed-size chunks to 89%⁴. A February 2026 benchmark across 50 academic papers put recursive, structure-aware 512-token splitting at 69% accuracy, 15 points above semantic chunking, because semantic chunking's small average fragment size, around 43 tokens, destroyed context¹.

There is no universal best strategy; it depends on your document types and query patterns¹. As a default, use recursive, structure-aware splitting that respects paragraph and sentence boundaries before falling back to character count¹. Use hierarchical parent-child chunking when a small retrieved fragment needs the full section around it to be interpretable⁴. Use proposition chunking, atomic factual statements, when users ask specific factoid questions and you need maximum precision, accepting that it costs an LLM call to generate¹. For short, self-contained documents like support tickets and FAQ entries, do not chunk at all; embed the whole document¹.

Whatever you choose, prefix each chunk with the document title and the heading path, and store stable chunk IDs. Queries often match the conceptual frame in headings better than body text, and you cannot do reliable evaluation without tracking a chunk across ingestion updates³.

Hybrid retrieval and reranking close the recall gap

Dense vector search is excellent at semantic similarity and weak at the things enterprises actually query on: exact identifiers, SKUs, ticket numbers, negation, and constraints³. Pure keyword search handles exact terms and misses paraphrases⁴. Hybrid retrieval runs both at once, dense and BM25, and fuses the results with reciprocal rank fusion, which fixes the vocabulary-mismatch problem by combining semantic and lexical signals⁴.

The measured lift is substantial. In a production case documented on Towards Data Science, pure dense retrieval scored 0.61 context precision and 0.82 faithfulness. Adding hybrid search at a balanced dense and lexical blend raised context precision to 0.71 and faithfulness to 0.85. Adding a cross-encoder reranker on top raised context precision to 0.79 and faithfulness to 0.89⁵. Reranking does not help recall, because it reorders the set you already retrieved. It turns recall into precision, which is exactly what a crowded context window needs⁵.

A production two-stage pattern is to retrieve 50 to 200 candidates cheaply, then rerank down to the 5 to 12 that actually reach the model³. A cross-encoder reranker, which scores query and passage jointly, is the quality default for queries with constraints and multi-part conditions. Late-interaction models like ColBERT are a cheaper middle ground. An LLM-as-reranker is a last resort, because it is slow and expensive³.

Measure retrieval separately from generation

Teams validate what the model says, not what it sees². They score the final answer, and the final answer can look excellent while the underlying retrieval is quietly failing. Deepchecks names the failure modes: the almost-relevant trap, where retrieval returns on-topic documents that lack the facts to answer, and coverage collapse, where retrieval captures only part of the required context and the model fills the gap with confident assumptions². Answer-quality metrics pass in both cases because the answer reads well and the citations look reasonable².

The fix is to evaluate the two layers independently⁴². Retrieval metrics include precision at k, recall at k, mean reciprocal rank, and the RAGAS context-precision and context-recall scores that ask whether the retrieved passages contain the needed facts and whether the set is polluted with noise⁴. Generation metrics include groundedness, whether the answer reflects the retrieved context, and faithfulness, whether it represents that context accurately⁴. If you only track one, you cannot tell whether a regression is a retrieval problem or a generation problem.

Build a golden set from real user queries, not clean synthetic ones. Fifty to one hundred manually curated queries give a meaningful signal, and below fifty, metric variance hides real improvements¹. Then keep evaluating after launch, because documents change, embeddings get swapped, and ranking logic evolves, and regressions show up in user complaints long before a static test suite catches them⁴. The tracing you wire up for the retrieval and generation stages is the same pattern we covered in our look at agent observability with OpenTelemetry.

A living system: drift, freshness, and knowing when to abstain

A RAG system degrades over time even when nothing about it obviously breaks. Embedding drift has three causes: model drift, when you switch embedding models without reindexing the corpus; corpus drift, when new document types reshape the vector space and pull in unrelated queries; and query drift, when user vocabulary shifts past what the embeddings were trained on⁴. The counter is versioning and observability: know which embedding model built each index, what chunking rules created each chunk, when a document was last re-ingested, and which retriever and reranker served a given request⁴.

Freshness is the quiet sibling. A document updated last month that your index has not reprocessed returns outdated answers with full confidence¹. Engineer it explicitly: hash document content at ingestion to detect updates, re-index only the changed documents, and remove chunks when access is revoked¹. Treat index staleness as a measured metric with an SLA, and for contradiction-heavy corpora, prefer the fresher source and say so when two retrieved chunks conflict¹.

Finally, give the system permission to abstain. When evidence is missing, stale, or contradictory, the correct answer is often "I do not know," and a system that says so builds more trust than one that produces a confident answer on weak evidence⁴.

Adroit on the Ground: the same failure, at a smaller scale

This is not academic for us. The Fortress, our own delivery stack, runs a knowledge base and a fleet of specialist agents, and we hit the retrieval failure mode routinely: an agent pulls the wrong note, or a stale version of a policy, and generates a confident answer on it. The disciplines above are the ones we actually practice, at honest scale. Our prompts and handoff contracts are versioned and reviewed, and a mechanical gate checks each draft against the source it claims to cite, which is the generation-side version of a faithfulness check. The same instinct, verify what the model saw rather than what it said, is why we treat retrieved context as something to check, not trust. The lesson generalizes: if you cannot trace an answer to the exact chunk the model read, you are flying blind.

The fix is earlier, and it is measurable

Demos reward systems that sound right. Production rewards reliability⁴. The reliable RAG system is the one where chunking preserves structure, retrieval combines semantic and lexical signals, a reranker curates the context window, and evaluation scores retrieval separately from generation, before and after launch. None of that is glamorous, and all of it beats another round of prompt tuning. When your RAG system degrades this quarter, do not ask which model to switch to. Ask whether the answer-bearing passage is reaching the context window at all.