Retrieval That Actually Helps
Why most retrieval pipelines underperform, and the fixes that matter: chunking, hybrid search, reranking, citation discipline.
The failure is usually retrieval, not generation
When a RAG answer is wrong, teams tend to blame the model. Measure first: check whether the correct passage was in the retrieved context at all.
- Was the passage retrieved? No → retrieval problem.
- Was it retrieved but ignored? → context/prompting problem.
- Was it retrieved and the answer still wrong? → then look at the model.
Most diagnosed “hallucinations” turn out to be case one.
Chunking decisions that change results
Chunk size is the highest-leverage knob, and there is no universal value. The right size depends on what a question needs as its minimum unit of evidence.
- Q&A over facts: small chunks (200–400 tokens), one idea each.
- Procedural docs: keep the whole procedure intact — splitting steps apart destroys the answer.
- Legal / policy: chunk by clause, never mid-sentence; add the section heading into every chunk’s text.
That last point generalizes: prepend structural context (title, heading path) to each chunk. A chunk saying “the deadline is 30 days” is useless without knowing which policy it came from.
| Document type | Chunk boundary | Why |
|---|---|---|
| FAQ | One Q&A pair | Self-contained by nature |
| API reference | One endpoint | Otherwise params get separated from description |
| Narrative prose | Paragraph, 10–15% overlap | Preserves flow |
| Code | Function or class | Syntactic unit, imports included |
Hybrid search beats either method alone
- Vector search captures meaning and paraphrases, but misses exact identifiers — error codes, product SKUs, names.
- Keyword/BM25 search nails exact tokens, but fails on synonyms and conceptual queries.
Run both and merge. Even a naive union beats either alone in practice; score fusion (RRF) is better and simple to implement.
If users search for things like ERR_2049 or Model-3B, keyword search is not optional.
Rerank before you generate
Retrieval at top-k=20 is cheap; reranking those 20 to a precise top-5 substantially improves answer quality. The pattern:
- Retrieve 20–50 candidates (fast, recall-oriented)
- Rerank with a cross-encoder or LLM scoring relevance
- Feed only the top 3–5 into generation
This costs one extra call and fixes a large share of “wrong passage retrieved” failures.
Force citation, then verify it
Require the answer to cite a chunk id for each claim. Two benefits: users can verify, and you can programmatically check whether cited chunks actually contain the claim.
Ask for, then spot-check with a grader: does chunk X support sentence Y? Answers that consistently cite retrieved chunks are far more trustworthy than those that paraphrase from memory.
What to measure
| Metric | What it tells you |
|---|---|
| Recall@k | Did the right chunk make it into candidates? |
| Citation accuracy | Do cited chunks actually support claims? |
| Answer-with-no-context rate | How often retrieval returns nothing useful |
| Chunk utilization | Are retrieved chunks actually referenced? |
High recall but low utilization means you are stuffing context with junk — cut k rather than adding more.