Every demo of retrieval-augmented generation (RAG) looks the same: load a PDF, embed it, ask a question, applaud. Then the same architecture meets a real corpus - tens of thousands of documents in three languages, half of them scanned contracts, a quarter of them stale - and answer quality collapses. The model is rarely the problem. In almost every production RAG system we have built or rescued, the failures live in retrieval: what got indexed, how it was chunked, what the query actually matched, and who was allowed to see it.
The uncomfortable truth is that RAG is an information-retrieval problem wearing an AI costume. Teams that treat it that way - with relevance metrics, index hygiene, and access control designed up front - ship systems that survive contact with users. Teams that treat it as prompt engineering ship demos.
Chunking Is a Data Modeling Decision
Fixed 512-token windows are a default, not a strategy. How you split documents determines what your retriever can ever return, so chunking deserves the same care as a database schema.
- ▸Structure-aware splitting: Split on headings, sections, and clauses rather than raw token counts. A contract clause cut in half is worse than useless - it retrieves confidently and answers wrongly.
- ▸Tables and figures need their own path: Serialize tables to markdown or key-value text and index them as separate chunks with a pointer back to the parent document. Naive extraction turns tables into word soup that embeds poorly.
- ▸Attach metadata at ingestion: Language, document type, effective date, owning team, and ACL group. Cheap to add now, nearly impossible to retrofit across millions of chunks later.
- ▸Parent-child retrieval: Embed small chunks for precision, but return the surrounding section to the model for context. Most mature stacks converge on this pattern.
Hybrid Retrieval Beats Any Single Index
Dense vectors are good at paraphrase and terrible at part numbers, error codes, and names. Lexical search with BM25 is the opposite. Production systems need both.
Run dense and lexical retrieval in parallel and merge with reciprocal rank fusion - OpenSearch, Elasticsearch, Qdrant, and pgvector-based stacks all support this pattern now. Then rerank the merged top 50 with a cross-encoder (bge-reranker or a hosted reranking API) down to the 5-10 chunks that actually enter the prompt. The reranker is routinely the single highest-leverage component in the pipeline: it costs tens of milliseconds and often lifts answer quality more than switching to a larger generation model.
Query handling matters just as much. Rewrite conversational follow-ups into standalone queries before retrieval, and route obviously structured questions - dates, IDs, aggregate counts - to SQL or an API instead of pretending vector search can answer them.
Evaluate Retrieval Before You Blame the Model
You cannot tune what you do not measure, and end-to-end reports that the answer feels wrong are unactionable. Split evaluation in two:
1. Build a golden set of 100-300 real user questions with the document passages that should answer them, sourced from support tickets and subject-matter experts. 2. Measure retrieval alone: recall at k and nDCG against the golden set. If the right passage is not in the top 10, no prompt will save you. 3. Measure generation separately: groundedness (does every claim trace to a retrieved chunk) and answer relevance, using an LLM-as-judge with spot-checked human agreement. 4. Wire both into CI so a chunking change or embedding-model upgrade cannot silently regress recall. 5. Log every production query with retrieved chunk IDs, so bad answers can be replayed and diagnosed offline.
This sequence turns arguments about model choice into ten-minute experiments with numbers attached.
Permissions and Freshness Are Not Optional Extras
A RAG index is a copy of your documents with the access model stripped off - unless you deliberately preserve it. Filter by ACL metadata at query time, resolved against the caller's identity, and treat any design that filters inside the prompt as a security incident waiting to happen. The same applies to deletion: when a document is removed or a GDPR erasure request lands, its chunks and embeddings must go too, which means you need document-to-chunk lineage from day one.
Freshness is the quieter failure mode. Stale indexes erode trust faster than wrong answers, because users notice when the system quotes last quarter's policy. Incremental ingestion tied to source-system webhooks or change-data-capture keeps the index honest; a nightly full rebuild is a fallback, not an architecture.
Where the Business Value Lands
None of this is glamorous, and that is precisely the point: the teams winning with RAG in 2026 are the ones doing retrieval engineering while competitors swap models. A pipeline with measured recall, hybrid retrieval, enforced ACLs, and lineage-aware deletion is not just more accurate - it is auditable, which is what turns an internal experiment into a system that legal, security, and compliance will let you put in front of customers. Invest in the boring layer first. The model at the end is the easiest part to replace.