For eight months, every change I made to my retrieval system made the benchmark better.
Then I ran the same queries through Amazon Bedrock Knowledge Bases on its default settings. It beat me.
That was useful, because I had no intention of deploying Bedrock. I went through the failures one by one. Of the 30 queries a Bedrock arm answered and mine missed, 22 had already retrieved the page containing the answer. My benchmark had been blaming search for a bug that happened after search. The other eight never retrieved the page at all, and they pointed at something I had never benchmarked: I was embedding entire PDF pages as single vectors.
Four cents of Bedrock found two bugs that eight months of grading myself had not.
TL;DR: Your own benchmark inherits your architecture’s assumptions, so it cannot test them. Running the same graded queries through a managed service you will never deploy costs a few cents and surfaces what your scoring was never set up to question. Mine found two bugs: an excerpt picker discarding answers off pages it had already retrieved, and one embedding per page hiding short answers inside long ones.
The benchmark that kept saying yes
I maintain pdf-mcp, an open-source MCP server: agentic RAG over your PDFs, one file or a whole folder, as a single MCP tool. The agent decides when to search; pdf-mcp does the retrieval and hands back excerpts. The design that shipped fuses per-document SQLite FTS5 rankings with cached embeddings. It is graded against hand-labeled evidence spans across a 100-document corpus, scored on whether the returned text actually contains the labeled answer.
The queries are split into classes by failure mode. The one that matters most here is described queries, which describe a concept without ever naming it. There is no keyword to match, so retrieval has to work on meaning alone.
Here is the first run where something other than my own code did the grading.
Run 1 of 2: initial 89-query diagnostic
| class | pdf-mcp | Bedrock defaults | Bedrock tuned |
|---|---|---|---|
| described | 0.120 | 0.360 | 0.400 |
| needle | 0.429 | 0.714 | 0.714 |
| spread | 0.640 | 0.600 | 0.680 |
| trap | 0.520 | 0.840 | 0.880 |
Two classes lost with confidence intervals excluding zero, so those gaps were not noise. A service where AWS picked the parser, the chunker, the embedding model and the retrieval path beat the thing I had been tuning.
My benchmark had reported steady improvement that whole time, and it was telling the truth. It simply had no way to show me the improvement was bounded.
Which left the only question worth asking: why did Bedrock win those queries?
Same budget, or it is not a comparison
Cap every arm at the same token count, not the same k. That one decision is what makes the comparison mean anything. An LLM has a context window, not a results limit, so tokens are the resource being spent. At an equal 2,000-token budget, pdf-mcp returned up to 25 short paragraph excerpts while the tuned Bedrock arm returned about 1.6 long reranked chunks. Both spent the same context. Capping at equal k would have handed pdf-mcp several times the budget and taught me nothing.
The two are the same job with different plumbing: Bedrock is RAG you provision, pdf-mcp is agentic RAG the agent carries with it. That is what makes the comparison fair and the token budget the right unit. I ran two Bedrock arms rather than one, because a managed service is a configuration surface and picking a single setting would have made it a strawman. The harness and the CDK stack are in the repo, along with the graded results.
That four cents deserves a line of backing. The knowledge base sits on Amazon S3 Vectors rather than OpenSearch Serverless, so no provisioned floor accrues while the stack is idle. Four cents is the query cost of the run itself, retrieval priced at $2.50 per million query requests and Cohere Rerank at $2.00 per thousand queries on the tuned arm only, with ingest modeled at another three to five cents. The retained stacks cost about two cents a month until I tore them down.
22 of 30 misses were not retrieval failures
For every query pdf-mcp missed, I checked whether the graded page had actually been returned, then re-extracted that page from the source PDF and re-checked the answer span independently of what the excerpt said.
Those 30 are the queries a Bedrock arm answered and pdf-mcp did not. A further 21 of the 89 were answered by no arm at all, so they say nothing about the difference between us and are excluded here.
Of those 30, 22 had already returned the correct page with the answer sitting on it.
| miss type | what retrieval did | what came back |
|---|---|---|
| picker miss | returned the right page | the wrong passage on it |
| routing miss | never returned the page | nothing to show |
These are two unrelated bugs wearing the same score, and the split was not even:
| class | misses | picker miss | routing miss |
|---|---|---|---|
| needle | 5 | 5 | 0 |
| trap | 10 | 10 | 0 |
| spread | 5 | 3 | 2 |
| described | 10 | 4 | 6 |
On needle and trap, doc-hit@3 (the right document in the top three) was 1.000. Routing was perfect. Every loss happened after the right page was in hand, when the code chose which passage of it to show. Better embeddings would not have touched it.
On described, six of ten misses never returned the page at all. Opposite bug, opposite fix. A single averaged score hides this completely, which is the argument for grading by class and never averaging across them. It is the same shape as a retrieval system with high recall that still answers wrong: the aggregate looks like one problem and is actually two.
One vector per page
The routing half had an embarrassing explanation.
pdf-mcp embedded one vector per page. Mean page length in that corpus is 721 tokens. Bedrock’s default chunking is roughly 300. A 15-token answer sentence was being averaged into a page-level vector alongside 700 tokens of unrelated text, which is exactly where paraphrase queries break, because no keyword hit exists to rescue the match.
When both 300-token chunks and 1,000-token chunks beat my page-level vectors, the obvious variable to test next was not the exact chunk size. It was whether one vector per page was too coarse.
I had run an embedding investigation before this. It compared model after model and held chunking at one vector per page throughout, because that was the architecture and I was tuning inside it. Choosing between search strategies is a decision you know you are making. Chunk granularity was one I made once, silently, and never revisited.
Sub-page embedding windows with a whole-page floor lifted described span recall from 0.120 to 0.240, enough that the gap against Bedrock’s defaults stopped being distinguishable from zero. The picker half became per-query excerpt routing: paragraph, window, or snippet, chosen by how many documents match the keywords.
Run 2 of 2: current 184-query benchmark
The last two columns are span recall differences, pdf-mcp minus each Bedrock arm, and a row is a win only when the 95% paired bootstrap confidence interval excludes zero. A tie means the interval includes zero, so the point gap could be query sampling noise.
| class | n | pdf-mcp | Bedrock defaults | Bedrock tuned | vs defaults | vs tuned |
|---|---|---|---|---|---|---|
| described | 83 | 0.229 | 0.301 | 0.253 | tie | tie |
| needle | 31 | 0.839 | 0.484 | 0.645 | +0.355 | +0.194 |
| spread | 45 | 0.711 | 0.422 | 0.511 | +0.289 | +0.200 |
| trap | 25 | 0.840 | 0.840 | 0.880 | tie | tie |
Two confirmed wins, two ties, no confirmed loss. This is a larger query set than Run 1, so the improvement is not the arithmetic difference between the two tables; the fix was measured on the original 89 with the Bedrock arms held fixed.
One caveat that cuts against me: containment quietly favors whichever arm returns fewer, longer, contiguous chunks, and the tuned Bedrock arm returned about 1.6 units per query where mine returned up to 25. Part of its margin is a property of the ruler, not the retrieval. I also kept the 54 of the 184 that no arm answered as zeros rather than dropping them, after checking each against the page images.
Benchmark against something you did not build
I was measuring carefully inside an assumption I had stopped being able to see. More rigor would not have surfaced it, because the assumption was part of the measurement itself.
An outside implementation breaks that cheaply. Stand up a managed service you have no intention of deploying, run your graded queries through it at an equal token budget, and score per class. Then resist the urge to tune it until it loses. Defaults are the point: they are the number a team with no knowledge of your corpus would get, and beating a strawman teaches you nothing.
You pay once, too. My harness caches the returned rows and reproduces the committed confidence intervals byte-for-byte without touching AWS again, so four cents buys a comparison I can re-run against every future change.
There is a version of this post where I benchmarked my project against AWS and won. It would have been a worse post, and the next eight months would have been worse. If your evaluation only ever compares you to yourself, most of what it measures is momentum.
