Most RAG advice is a component review. Which vector database. Which embedding model. Which retrieval framework. I spent a year building retrieval for AI agents and none of my real quality problems lived in those choices. They lived one level up, in six decisions I made early, half of them by accident, and paid for later.
I didn’t design these six decisions upfront. Each one surfaced as a failure: an agent burning its tool budget walking pages, a fusion step quietly demoting answers that were already ranked first, truncated excerpts forcing a retrieval step that shouldn’t exist. This is the difference between this guide and a generic RAG explainer. Every decision below earned its place by breaking something.
This guide is the map I wish I’d had. Each decision below is stated tool-agnostically, comes with the default I’d pick today, and links to the benchmark write-up that settled it. The numbers come from retrieval systems I built and measured, mostly over real PDF corpora. Your corpus will differ. The decisions won’t.
TL;DR: Most RAG failures don’t come from picking the wrong vector database. They come from six earlier decisions: whether to retrieve at all, which search mode, fusion vs routing, chunk granularity, whether document structure survives extraction, and who drives retrieval, your pipeline or the agent. Granularity is the sleeper: changing it flips the right answer to the ranking question. Benchmark-backed defaults for each.
The decision table
| # | Decision | Options | My default |
|---|---|---|---|
| 1 | Retrieve at all? | prompt / RAG / fine-tune / long context | Prompt first, retrieve when knowledge is missing |
| 2 | Search mode | keyword / semantic / hybrid | Hybrid at page grain |
| 3 | Combine modes | fuse (RRF) / route per query | Fuse |
| 4 | Granularity | page (fixed chunk) / section | Section, and re-test ranking after switching |
| 5 | Structure handling | raw extraction / structure-aware | Structure-aware, verified per layout |
| 6 | Who drives | pipeline RAG / agent-navigated | Agent-navigated for documents |
The order matters. Each decision constrains the next, and the expensive mistakes come from answering a later question while silently assuming an earlier one:
That arrow from Decision 4 back to Decision 2 is the one that cost me the most. We’ll get there.
Decision 1: Should you retrieve at all?
Retrieval is only one of four ways to give a model knowledge, and it’s the most expensive of the cheap ones: a pipeline, a failure surface, and a latency floor. RAG is the default reflex in 2026, and it is frequently the wrong one.
My working order: prompt engineering first, retrieval when the model demonstrably lacks knowledge it needs, fine-tuning only for behavioral consistency that prompting can’t hold. I wrote up the full decision framework in RAG vs Fine-Tuning vs Prompting, and it has aged well: most teams still over-invest in the heavy options before exhausting the cheap one.
The 2026 wrinkle is long context: million-token windows make inlining a mid-sized corpus a real alternative. It wins on simplicity for one-shot analysis, loses on per-request cost and attention. I haven’t benchmarked that trade yet; the numbers slot in here when I have them.
Default: exhaust prompting first. Retrieve when knowledge is missing, changing, or too large to inline.
Decision 2: Keyword, semantic, or hybrid?
The mistake here is treating this as a technology choice. It’s a query-type choice.
Semantic search wins when the query and the content don’t share words: a query for “income growth” needs to find a paragraph that says “revenue increased.” Keyword search wins when precision is the point: invoice numbers, part codes, exhibit references. INV-2024-00847 has no synonyms, and an embedding model will happily return you a different invoice.
Agent queries are a mix of both, and you usually can’t predict which kind arrives next. That’s the case for hybrid as the default: run both, combine the results. The full breakdown of when each mode wins, with the failure cases, is in Semantic vs Keyword Search for AI Agents.
Default: hybrid at page grain. The exception: if your query stream is all identifiers (ticket lookups, SKU searches, citation retrieval), keyword alone is simpler and strictly better. And hold the hybrid answer loosely until Decision 4, because grain changes it.
Decision 3: One retriever or a router?
Decision 2 left you running two search modes, which creates the next problem: something has to combine their answers. Two candidate architectures: fuse every query through both modes and merge the rankings (Reciprocal Rank Fusion), or put a router in front that classifies the query and picks a mode.
Routing sounds smarter. It benchmarked worse. Across seven agentic scenarios on real PDFs, hybrid RRF scored a mean reciprocal rank of 1.00 while the query router scored 0.67, and the router’s misses clustered exactly where routing was supposed to help: ambiguous queries that straddle lexical and semantic intent. The router misclassified precisely when classification was hard, which is precisely when it mattered. Fusion cost 4ms. The full benchmark is in Hybrid Search vs Query Routing in RAG.
The general lesson: a component that must be right about a hard judgment to add value will fail on the inputs where the judgment is hard. Fusion doesn’t judge. It hedges.
Default: fuse, don’t route. Spend the routing complexity budget elsewhere. Routing becomes attractive only when every retrieval call has a real cost: metered external search APIs instead of two local indexes. Then the classification gamble can be worth what it saves.
Decision 4: What granularity?
This is the decision nobody puts on the whiteboard, and it’s the one that quietly rewrites the other answers.
Fixed-size or page-level chunks are what every tutorial reaches for. But documents aren’t organized in pages, they’re organized in sections, and agents ask questions about sections. When I measured what page-grain retrieval costs an agent on a dense document, the answer was 5.79 extra read calls per query on one paper, with 11 of its 24 sections completely unrecoverable inside a 10-call tool budget. The agent wasn’t retrieving, it was walking pages trying to reassemble what chunking had cut apart. Section-aware retrieval delivered the same content in one call (the benchmark).
Then the trap: after switching to section grain, I assumed the Decision 2 answer still held. It didn’t. Section titles are lexical anchors, so at section grain plain BM25 hit 0.93 MRR on lexical queries while adding semantic fusion dragged the same queries down to 0.63 (why fusion breaks at section grain).
Changing the chunk size changed the winner. The hybrid setup that was correct at page grain became a 33% regression at section grain. The ranking algorithms stayed the same. Only the unit of retrieval changed. That’s the backward arrow in the diagram, and it’s why these six decisions are a sequence, not a checklist.
Default: section grain for structured documents. Page or fixed-size chunks remain reasonable when documents have little internal structure: logs, chat transcripts, email threads. Either way, re-run your search-mode benchmark after any granularity change, because the decisions interact.
Decision 5: Does the document’s structure survive extraction?
Decisions 2 through 4 all share one assumption: the text you indexed is the text the document contains. For real-world documents, that assumption fails silently. And when it fails, nothing downstream can compensate: if extraction scrambles structure, no ranking algorithm can unscramble it.
First, excerpt boundaries. Search results that return truncated snippets force the agent into a second retrieval step to recover context. Switching to excerpts that respect paragraph boundaries moved answer containment from 80% to 97% on a 30-query benchmark, and the dominant agent pattern collapsed from “search, then read pages, then answer” to “search, then answer” (how one search change eliminated an agent step).
Second, reading order. Multi-column layouts interleave sentences from different columns when extracted naively, and vertical scripts break left-to-right assumptions entirely. I’ve benchmarked and shipped fixes for both (two-column extraction fidelity went from 0.564 to 0.816 in the multi-column work); those write-ups land on this blog over the next few weeks.
Default: structure-aware extraction, verified per layout family in your corpus, before you tune anything else. Born-digital single-column documents mostly survive naive extraction; this decision is really about scanned, multi-column, and non-Latin layouts.
Decision 6: Who drives retrieval, you or the agent?
Every decision so far assumed retrieval is something you design in advance. This last decision questions that assumption. Classic RAG is a pipeline: embed the query, fetch top-k, stuff the context, generate. The agent never knows retrieval happened. That design assumes you can pick the right k and the right query formulation before seeing the question. For document work, agents given retrieval as tools beat pipelines that hide it, because the agent can adapt: scout the structure first, search within what it learned, read only what survived.
That scout-then-read behavior wasn’t something I designed. Agents converged on it themselves once the tools made it possible, and it’s the backbone of the five navigation patterns that survived production across 32,000 downloads of my PDF server (how AI agents should read PDFs).
The boundary on the other side: retrieval tools are for knowledge, not for facts that have an authoritative source. An agent that answers numeric questions from search results is guessing from headlines; give it the API instead (why agents need APIs, not search).
Default: for document corpora, expose retrieval as tools and let the agent navigate. Reserve pipeline RAG for high-volume, low-variance queries where per-call agent reasoning is too expensive.
What I’d revisit at scale
Honest limits of this guide. My benchmarks run on document corpora in the gigabytes, not terabytes; at warehouse scale, index cost and freshness dominate in ways my numbers don’t capture. The long-context trade in Decision 1 is stated from reasoning, not measurement, and it’s the next benchmark on my list. And every corpus above is documents; if your retrieval target is a database or an event stream, Decision 6 collapses toward APIs and most of the middle decisions dissolve.
One more thing about all those benchmark citations. None of these defaults came from a public leaderboard. Every one came from a benchmark over my own corpus, and that’s the point: retrieval is unusually corpus-dependent. The section-grain BM25 result exists precisely because my documents have strong section titles; a corpus without them could flip that answer again. Don’t copy my defaults blindly. Copy the practice of benchmarking each decision on your own data before committing to it.
What I’d keep at any scale: the ordering. Decide whether to retrieve before how, decide grain before ranking, and verify extraction before trusting any number the ranking benchmarks produce. The expensive failures in agent RAG are rarely a bad component. They’re a good component answering the wrong question. Retrieval quality is not mostly determined by your embedding model or your vector database. It’s determined by the architectural decisions you make before those components ever matter.
