Retrieval-augmented generation is the most requested AI capability we are asked to build, and the one most likely to disappoint after launch. The demo is always impressive. Someone loads twenty clean PDFs, asks three well-chosen questions, and the answers are excellent. Six weeks later the same system is running against 400,000 scanned documents of wildly varying quality, and the answers have quietly become unreliable.
The gap between those two states is almost never the model. It is everything around it.
Retrieval quality is the whole game
If the right passage does not reach the model, no amount of prompt engineering will save the answer. Yet retrieval is consistently the least-examined part of the pipeline.
Three things move the needle more than anything else:
Chunking that respects document structure. Splitting on a fixed character count is the default in every tutorial and it is almost always wrong. A table split across two chunks becomes meaningless in both. Chunk on semantic boundaries — sections, clauses, list items — and carry the parent heading into each chunk as context.
Hybrid search rather than pure vectors. Dense embeddings are excellent at conceptual similarity and surprisingly poor at exact identifiers. If a user searches for invoice INV-2024-8871 or a specific policy code, lexical search finds it instantly and vector search may not find it at all. Run both and fuse the results.
A reranking pass. Retrieve generously — say the top fifty candidates — then rerank down to the five you actually pass to the model. A cross-encoder reranker is far more accurate than the initial retrieval step, and running it over fifty candidates rather than the whole corpus keeps it affordable.
Grounding is a product requirement, not a nicety
Any system that produces an answer without showing where it came from will eventually produce a confident, well-written, entirely wrong answer. In a healthcare or financial context, that is not a bug to fix next sprint.
Every generated claim should carry a citation back to the source passage, and the interface should make that passage one click away. This does two things: it lets users verify anything that looks surprising, and it changes their relationship with the tool. People trust a system that shows its work, and they learn its limits quickly.
We also route low-confidence extractions to human review rather than surfacing them. A system that says "I am not sure, here are the two passages that seem relevant" is more useful than one that guesses.
You cannot improve what you do not measure
The single biggest difference between teams whose RAG systems improve and teams whose systems drift is an evaluation set.
Build one early. A hundred question-and-answer pairs written by actual domain experts is enough to start. Run it on every change to the prompt, the chunking strategy, the retrieval parameters or the model version. Track retrieval recall separately from answer quality, because when a number drops you need to know which half of the system moved.
Without this, every change becomes a matter of opinion. Someone tweaks a prompt, tries two questions, decides it feels better, and ships a regression that nobody notices for a month.
Cost discipline from day one
Token costs scale with usage in a way that surprises finance teams. A few habits keep it manageable:
- Cache aggressively. Identical or near-identical queries are far more common than you would expect, particularly in internal tools.
- Route by difficulty. Not every query needs your most capable model; a smaller model handles straightforward lookups at a fraction of the cost.
- Trim the context. Passing ten chunks when five would do doubles your input cost and frequently makes answers worse, not better.
- Set per-tenant budgets and alerts before launch, not after the first unexpected invoice.
Start narrow
The strongest predictor of success we have seen is scope discipline. Teams that pick one document type, one user group and one question category get to production and then expand from a working foundation. Teams that try to build a universal assistant across the entire organisation tend to produce something that is mediocre at everything.
Pick the narrowest problem that is genuinely worth solving. Make it excellent. Earn the right to expand.
Considering a retrieval or copilot project? Talk to our AI engineering team — we will tell you honestly if a simpler approach would serve you better.