Ask a folder of 100 PDFs one question and the hard part is merging, not searching: every document is happy to rank its own pages, and combining those 100 private rankings into one honest list is the part most multi-document RAG tutorials skip. It is also where the first design everyone reaches for breaks.
This post is the version that shipped: one query across a folder of 100 PDFs, one correctly ranked hit list, no vector database, no framework, everything in SQLite. I build pdf-mcp, an open-source MCP server for large-PDF workflows I use daily through Claude Desktop and Claude Code. For most of its life it read one document at a time. pdf_corpus_search is what happened when I stopped pretending that was enough.
TL;DR: pdf-mcp’s corpus tools search a 100-PDF folder in one sub-second call, no vector database: per-document SQLite FTS5 plus cached embeddings, fused with Reciprocal Rank Fusion. BM25 scores do not merge across per-document indexes; ranks do, and tied ranks need a content-based tie-break. Hybrid mode put a gold document in the top 3 on 89.9% of 89 graded queries over a 100-paper corpus, and on 81.8% over near-duplicate annual filings. The misses concentrate in one class, questions that describe a concept without naming it, where the rate drops to 72% and fusion buys nothing over semantic alone.
One tool call, 100 documents
Point an AI agent at a directory of PDFs and ask “which of these discusses retry backpressure?” and you get to watch it cope. It opens one document. Searches it. Opens the next. Twenty tool calls later it has burned its context window on tables of contents, and the answer was in document 61. The fix wants to be one call: search everything, return one ranked list.
Here is the shipped version of that fix: pdf_corpus_search pointed at a 100-document, 2,238-page corpus of arXiv papers (the same corpus the benchmark below uses), on my laptop against a warmed cache. Real output, trimmed:
query: "Thomson problem spherical crystals"
0.40s | hybrid | coverage: 100 of 100 docs
1. 0707.3690.pdf p.1 semantic 0.74
"...a sphere to minimize the repulsive Coulomb
potential (the Thomson problem) and experimentally
found in spherical crystals formed by self-assembled
polystyrene beads..."
2. 0707.3690.pdf p.6 semantic 0.69
"...formed in spherical crystals of mutually
repelling polystyrene beads self-assembled on
water droplets in oil..."
One call, under half a second, and the right paper’s two relevant pages are ranked 1 and 2 out of roughly 2,200 candidates. The next move is obvious and cheap: pdf_read_pages on the winner. The twenty-call flail becomes a two-call loop.
The agentic part is not the search. It is that the tool tells the agent the truth about its own coverage: every answer carries an explicit searched: 100 of 100, so partial results are visible instead of silent.
Notice what is missing from this scene: an ingestion pipeline. There is no chunking configuration, no vector store, no embedding job to babysit. The first call warmed everything into SQLite; editing one file invalidates that file’s cache and nothing else.
The rest of this post is how that ranked list gets built, because the first design I reached for was wrong in a way worth naming.
The score portability trap
Every document I cached already had its own SQLite FTS5 index with BM25 scoring, built for single-document search. So the first cross-document design that suggests itself is barely even a design:
for each doc: search(doc) -> hits with BM25 scores
merge all hits, sort by score, take top 10
It works in the demo. It is also quietly, structurally wrong.
BM25 is not a property of a page. It is a property of a page inside an index. The score is built from corpus statistics: inverse document frequency and length normalization computed over whatever that index happens to contain. Give every PDF its own index, and every score comes out denominated in a different currency. The term “backpressure” might be exotic inside a 12-page memo and mundane inside a 400-page operations manual, and their scores will say so, loudly, for reasons that have nothing to do with which page answers the question.
Sort those numbers together and nothing crashes. The ranking is simply wrong, in a way that surfaces in production as an agent confidently quoting the wrong document.
I call this the score portability trap, and the rule out of it fits in one line:
BM25 scores are not portable across indexes. Ranks are.
Once the trap has a name, the design space collapses to exactly two honest options:
- Make the scores comparable. Build one corpus-wide FTS5 index, so every score shares the same IDF statistics.
- Stop using scores. Keep the per-document indexes and merge by rank instead.
Neither option is new. Information retrieval has studied this for decades as results merging in federated search; Shokouhi and Si’s survey states it outright: “the document scores or ranks returned by multiple collections are not directly comparable.” A folder of 100 PDFs is federated search where every collection is one document.
Option 2 has an answer old enough to predate the entire RAG industry: Reciprocal Rank Fusion, from a 2009 information-retrieval paper. The intuition is voting: every document votes with the order of its hits rather than their scores, and order is the one thing per-document BM25 gets right. Each item contributes 1 / (k + rank) from every list it appears in, with k = 60. No normalization, no tuning, no model. The whole thing is shorter than its own docstring:
def rrf_fuse(rank_lists, k=60):
scored = []
for hits in rank_lists:
for rank, (doc, page) in enumerate(hits):
scored.append((1.0 / (k + rank), doc, page))
scored.sort(key=lambda t: (-t[0], t[1], t[2]))
return [(doc, page) for _, doc, page in scored]
Note what breaks ties in that sort: after the RRF score comes t[1], the document’s path. That looks like a harmless determinism detail. Hold on to it.
The paper settles only half the question, though. Rank fusion beats score fusion across incomparable rankers; it says nothing about per-document fusion versus one global index. The benchmark would have to decide that.
Why per-document fusion won
Which option wins is an empirical question, so I built both arms and benchmarked them on 100 arXiv papers (2,238 pages, 79 of them pure distractors). The ground truth at that point was 64 graded queries: hand-picked gold pages with relevance gains, split into three ways cross-document search fails.
- needle: the answer lives on one page of one document
- spread: relevant pages are scattered across several documents
- trap: a distractor document is lexically similar but wrong
The trap class is the global index’s home turf: honest corpus-wide IDF should recognize which of two lexically similar documents actually answers. Measured, that advantage came to +0.006 NDCG@10, a tie, and needle tied as well.
Spread did not tie, and it broke against the global index: 0.332 versus 0.381. The mechanism is the 79 distractors. One corpus-wide table ranks all 2,238 pages together, so on a query whose gold pages sit thinly across two or three documents, distractor pages flood the top 10 and bury them. Four spread queries collapsed by 0.49 to 0.61. Per-document fusion is structurally immune to that: it fuses each document’s own ranking, so every gold document’s best page competes for a slot no matter how many distractors exist. Fusion won overall, 0.547 to 0.531, on the strength of that class.
Cost was not close either. A corpus-wide index rebuilt per query ran 2.3 seconds, against per-document indexes that are already cached and invalidate one file at a time.
None of that showed up at small scale. An earlier 21-document spike picked the global index on a trap-class margin resting on two queries, and scaling the corpus overturned it. Small benchmarks have fooled me before (BM25 beat hybrid search in a matchup I expected to go the other way), so every number here comes from a run at full corpus scale. That rule, benchmark at the scale you intend to run, is item one in my RAG decision guide.
One more design decision hides inside the fusion itself. RRF gives every document’s best page the identical score, 1/(k + 0), so when many documents match, the entire top band is tied, and the innocent (doc, page) tie-break in the snippet above would rank that band alphabetically. A ranking that falls back to file paths is an ls. The shipped code breaks those ties with content instead: each tied document is scored by how many distinct query terms it carries, weighted by each term’s rarity across the matching documents. Document frequencies come from the documents the query just matched, so it costs no extra I/O and needs no corpus-wide index.
What shipped: two-stage fusion
The production tool runs per-document fusion twice.
Stage 1 fuses the per-document keyword rank lists into one cross-corpus keyword ranking. Stage 2 fuses that with a semantic ranking, the same hybrid pattern that works inside a single document, now operating across the corpus. Stage 1’s input rankings are disjoint: a page belongs to exactly one document, so contributions never add and the fusion degenerates into a round-robin. Watch the trap example pass through it:
memo.pdf own index: 1. p.4 2. p.2
manual.pdf own index: 1. p.210 2. p.33
fused by 1/(60 + rank), rank within its own document:
memo p.4 1/60 <- every doc's best page
manual p.210 1/60 <- shares the top band
memo p.2 1/61
manual p.33 1/61
The BM25 scores that caused the trap, memo’s overheated 7.8 against manual’s honest 2.4, are gone before the merge begins. A document’s second-best page can never outrank any document’s best, which is exactly the fairness the trap demands. Within a band, plain RRF has no opinion; the shipped code orders tied documents by the coverage signal from the last section rather than by their paths.
Stage 2’s two input rankings overlap: the same page can appear in both the keyword and the semantic ranking, and when it does, its contributions add:
keyword arm: 1. memo p.4 2. manual p.210 ...
semantic arm: 1. manual p.210 2. manual p.211 ...
manual p.210 1/61 + 1/60 = 0.0331 <- in both arms
memo p.4 1/60 = 0.0167 <- keyword only
manual p.211 1/61 = 0.0164 <- semantic only
Consensus is the whole mechanism: the page both arms agree on pulls away from pages only one arm likes. Stage 1 makes the merge fair; stage 2 makes it sharp.
The semantic arm is where “you need a vector database” quietly evaporates. A 100-document corpus is about 2,200 page-level vectors from bge-small, stored as blobs in the same SQLite file as everything else. Exhaustive cosine over 2,200 vectors is one NumPy matrix product. Approximate nearest-neighbor indexes solve a problem that starts around a million vectors; at corpus scale, brute force is not the compromise, it is the correct algorithm.
The query set has since grown to 89, with a fourth class the first three did not cover:
- described: the question uses the reader’s words, not the document’s, with no term in common (“does normalizing layer inputs converge in fewer training steps at equal accuracy”)
Measured on the shipped tool over the 100-document corpus and all 89 graded queries:
| Mode | NDCG@10 (page) | NDCG@10 (doc) | Gold doc in top 3 | s/query |
|---|---|---|---|---|
| keyword | 0.459 | 0.772 | 83.2% | 0.47 |
| semantic | 0.484 | 0.792 | 83.2% | 0.25 |
| hybrid | 0.541 | 0.838 | 89.9% | 0.47 |
The arms complement exactly as designed. Keyword anchors literal precision: 0.968 on needle queries. Semantic shrugs off the lexical traps that fool keyword search: 0.785 versus 0.596 at page level. Fused, they beat both, and the number an agent actually cares about, a correct document in the top 3, reads 89.9%: 80 of 89.
The two NDCG columns are the same rankings at two granularities. The ground truth grades two or three pages per query while a relevant document matches many more, so most page-level “misses” land on unlabeled pages of correct documents. Doc-level is the honest read, and the column that predicts the agent’s next move, pdf_read_pages on a document: hybrid 0.838, needle and trap at a perfect 1.000.
Where fusion stops helping
The overall row hides which queries the misses come from, and they are not spread evenly. Two of hybrid’s nine top-3 misses are spread queries, the class where the right move is fanning out per document anyway, and doc_match_counts exists to trigger exactly that. The other seven are all described.
That class is a quarter of the query set and it sets the page-level overall nearly by itself. Every mode scores 0.241 or below on it. Keyword manages 0.134, about what “no shared terms” predicts. And hybrid does not improve on semantic: 0.241 against 0.241 at page level, 0.698 against 0.698 at doc level, the same 72% top-3 rate. Every described number is identical, because a keyword arm with nothing to match contributes nothing to the fusion.
That is a limit of the design, and it is not a fusion problem. RRF reconciles rankings. It cannot manufacture a signal neither arm has, so when the question and the document share no vocabulary, the semantic arm is the whole system.
That ceiling is the embedding model, but only for queries phrased the way the benchmark phrases them. I tried the model side first. A live benchmark of four fast English fastembed models turned up no challenger that cleared the MRR-lift gate, so the default stayed. Phrasing is where the number moves. In a 78-call behavioral eval, a caller model saw only the tool signature and the shipped query docstring, and it rewrote these questions into terms of art without being told to. Those queries put a gold document in the top 3 on 88% of the class, against 72% for the raw question text, and single-hop answer recall went from 48% to 80%. So 72% is the floor for verbatim benchmark strings, not the number a real caller sees. What limits described queries is wording, and an agent reading the tool description is already most of the way past it.
The second corpus reaches the same place from a different direction. 24 public-company annual filings, 3,545 pages, eight companies across three fiscal years, every document a near-duplicate of two others. Hybrid still won overall on both axes, but doc-level NDCG@10 came in at 0.776 and a gold document reached the top 3 on 81.8% of queries. Its weakest class is concept, that corpus’s version of described, where hybrid again failed to beat semantic: 0.468 against 0.488. Different corpus, different name for the class, same result. Fusing lexical rankings buys nothing when the query has no lexical overlap to fuse on.
When you actually need a vector database
The honest boundary, because this stack has one.
| Skip the vector database if… | Reach for one if… |
|---|---|
| your corpus is tens to hundreds of documents, not millions of chunks | you are past roughly a million vectors and need ANN indexes |
| documents change independently, so per-doc cache invalidation matters | many writers update the index concurrently |
| the search runs on one machine (a laptop, a single server, an MCP server) | retrieval must be a shared network service with its own scaling story |
| exhaustive cosine over your vector count is sub-millisecond anyway | you need metadata filtering fused into the ANN search itself |
pdf_corpus_search caps its corpus at 100 files on purpose: that is the design boundary that makes the no-infrastructure version correct. Past it, the trade-offs genuinely change.
Try it on your own folder
Everything above runs from one pip install:
pip install pdf-mcp
claude mcp add pdf-mcp -- pdf-mcp
(Configs for Claude Desktop and other MCP clients are one JSON block each, in the README.)
Then point your agent at any folder of PDFs and ask the question you actually have: “which of these papers measures backpressure under load?”, “which contract mentions early termination?”. The first query warms the cache within its time budget and says so; every one after is sub-second. There is nothing to stand up and nothing to deploy. For the daily workflow these tools reward, one folder per topic, warm, catalog, query, see Turn a Folder of PDFs Into an AI Agent Knowledge Base.
Rank, don’t score
The multi-document RAG stack everyone reaches for, a vector database plus a framework plus an embedding pipeline, solves an infrastructure problem most corpora do not have. The problem a folder of 100 PDFs actually has is subtler: per-document relevance scores that look mergeable and are not.
Name the trap and the design follows. BM25 scores are not portable across indexes; ranks are. Fuse ranks with RRF, break the ties with content rather than with whatever your sort key happens to grab, keep everything in SQLite FTS5 plus a few thousand cached vectors, and the vector database becomes what it always should have been at this scale: unnecessary.
