I got a question recently that I could not answer. Agent harnesses keep getting better. Claude Code can now point subagents at a folder of PDFs, screen them in parallel, and read whatever looks relevant. So does a tool like pdf-mcp, an agentic RAG tool (a search index over your PDFs that the agent queries through tool calls), still matter? Or is saving tokens all it does now?
It is a fair question, and I wanted to know the answer too. I build pdf-mcp. The built-in reader used to have real gaps, and the tool existed to fill them. If the harness has caught up, I’d rather find out than assume.
So I tested it. I took questions I did not write, from a published dataset, and ran each one twice: once with pdf-mcp, once with Claude Code’s built-in file tools only. The short answer: on accuracy, the tool did not matter. On the work it took to reach the same answers, it mattered a great deal. Why comes down to how the harness finds papers in the first place.
TL;DR: On 53 multi-paper questions over 120 papers, Claude Code gave equally good answers with and without pdf-mcp. Without it, the agent usually found papers by reading page 1 of every file. With it, the agent read 7.5 pages per question instead of 187, answered nearly four times faster, and named the right section more often.
What “smarter search” looks like inside the harness
Does the built-in side search intelligently? It has no index, so its only way to find a paper is to open it.
So it opens them. On most questions it starts several subagents, splits the folder between them, and has each one read page 1 of every paper in its share. Then it reads further into the papers that look relevant. One built-in reply put it plainly:
I checked the first page of all 120 papers, and the only other microarray-classification paper (PMC1569876) doesn’t discuss these challenges, so the answer rests on one paper even though you expected several.
There is real intelligence here, in deciding what to read next and what to skip. The finding step itself is brute force, run in parallel. That matters for the question, because the cost of brute force grows with the folder and the cost of an index lookup does not.
This is a PDF problem more than a harness problem. On a folder of markdown or source code, Claude Code would grep for a term across all 120 files in one call and open only the hits. It cannot do that to a PDF, so it opens them instead. pdf-mcp exists to make PDFs searchable the way text already is, with ranking on top. If your source of truth is plain text, expect the cost gap below to be much smaller, and the case for a tool like this to rest on ranking and section pointers alone. I have not measured that.
The test
The questions come from DocHop-QA (Park et al. 2025), a published set of questions over open-access PubMed Central papers. Someone else wrote them, so I could not tilt them toward my tool. I kept only questions that need two papers to answer, filtered out unusable ones, and sampled until the folder held 120 papers: 62 questions over 951 pages. DocHop-QA ships no gold answers, so I wrote 3 or 4 short key points per question from the dataset’s own evidence to mark against.
Before running anything, I froze the questions, key points, prompts and rules in git. Both modes ran the same pinned model, claude-opus-5-5, each trial in a fresh headless claude -p session, with the same prompt apart from one paragraph about tools. Both could use subagents. That is how the built-in side screens a folder, so banning them would have rigged the test against it. I counted them as cost instead.
Each answer had to list its SOURCES. Evidence was scored from that list only: did it cite both right papers, and the right section of at least one? The answer itself was marked correct (y), partial, or wrong (n) against the key points.
Does it matter for accuracy? No
Nine questions dropped out for reasons covered further down, leaving 53 compared in both modes. “Lenient” scores y as 1, partial as 0.5 and n as 0.
| pdf-mcp | built-in | |
|---|---|---|
| Cited both evidence papers | 31 | 30 |
| Cited one of the two | 22 | 22 |
| Cited neither | 0 | 1 |
| Named the right section | 46 (87%) | 40 (76%) |
| Correct: y / partial / n | 6 / 47 / 0 | 5 / 47 / 1 |
| Lenient accuracy | 55.7% | 53.8% |
| Wall time per question | 25 s | 92 s |
| Pages read per question | 7.5 | 187.1 |
| Papers opened per question | 1.9 | 84.0 |
| Subagents per question | 0.0 | 4.2 |
On 47 of 53 questions the two modes got the same mark. Of the 6 that differed, 4 went to pdf-mcp and 2 to the built-in side. That is too few to call a winner. On answer quality, the premise behind the question holds: the harness does fine on its own.
The more telling number is the misses. pdf-mcp failed to cite one of the two right papers on 22 questions. The built-in side missed the same paper on 21 of those 22. Two completely different ways of finding papers, an index and a read-everything sweep, failed in the same places. Several of those papers sit among near-duplicates from the same workshop proceedings, where a sibling paper looks just as relevant. At this size, finding the papers was not the bottleneck. Deciding which ones answer the question was, and the same model did that deciding in both modes.
Same answers, very different work
Saving tokens was never in doubt. I measured pages read, not tokens. Pages read, by the main session and its subagents, are a rough stand-in for tokens spent, and pdf-mcp read about a twenty-fifth of them. Three other differences showed up.
The first is time. pdf-mcp answered in 25 seconds against 92, nearly four times faster. That is about a minute saved on every question. For one lookup, a minute may not matter much. In an agent workflow that searches the same folder again and again, the delay sits inside the loop. At these averages, ten questions in a row would take about four minutes with pdf-mcp and fifteen without it. I did not run a ten-question workflow; that is the per-question average multiplied out.
The second is knowing where the evidence sits. pdf-mcp named the right section 46 times against 40. Each search hit names a paper and a page, with an excerpt, so the model reads near the evidence instead of from the top. If you check the agent’s work, that helps: a citation that names the section is one you can go and verify. The gap held in two of three batches and tied in the first:
| Batch | Questions | Section right: pdf-mcp | Section right: built-in | Lenient: pdf-mcp | Lenient: built-in |
|---|---|---|---|---|---|
| 1 | 19 | 14 | 14 | 55.3% | 55.3% |
| 2 | 16 | 15 | 13 | 59.4% | 53.1% |
| 3 | 18 | 17 | 13 | 52.8% | 52.8% |
The third is subagents. The built-in side ran 4.2 per question and pdf-mcp ran none. Subagents are free to start, but each one reads pages too, and those pages count.
Where the answer could change: folder size
Here is the part one folder size could not settle. The built-in side opened 84.0 papers per question and pdf-mcp opened 1.9. In the smaller-folder test below, the first number grew with the folder and the second barely moved. If the same pattern held, the sweep would keep getting more expensive as the folder grew, while the index should stay close to the cost of answering the question. At some size, brute-force reading stops being merely inefficient and becomes the question of whether the sweep is practical at all.
To test that, I reran 10 of the questions on a folder of 30 papers: their 20 right papers plus 10 drawn at random.
Time followed the same shape. The built-in side more than doubled, from 44 seconds to 98, while pdf-mcp went from 23 to 26. The sweep got slower as well as heavier.
The method changed too. In these runs the harness showed no third strategy: at 30 papers it read the folder itself and started no subagents; at 120 it started 4 to 8 subagents on 7 of the 10 questions and opened every file.
Two folder sizes are two points, not a curve. The 30-paper folder was also two-thirds right papers, so this run says nothing about accuracy, only about reading cost. And I measured downward, not up to 1,000. The harness may screen a much bigger folder differently.
Why I trust these numbers, and where I don’t
Getting a fair answer took more care than I expected, and three things nearly broke the comparison.
My own tool nearly lost on a technicality. pdf-mcp 3.5.0 refuses a corpus search over more than 100 PDFs, a guard I had never tested beyond. On the first run, all 41 answered tool trials hit it. The model worked around the cap, usually by splitting the folder into two searches of 60, which my own error message warns against. In the dry run it did that without saying so in its answer. That run measured the workaround, not the tool. I re-ran every tool trial on a one-line build with the cap raised to 200, and none hit it. Every pdf-mcp number above is from the re-run.
The model I asked for was not always the model I got. Every trial pinned claude-opus-5-5. Reading the model id back from each transcript showed that some ran on claude-opus-5 instead, with nothing else looking wrong. It followed the question: seven questions drifted in tool mode on every attempt, six of the same seven drifted in built-in mode, and an eighth switched models partway through a built-in session. All 8 are out. One more, about immune responses in virus-induced asthma, was refused by the API’s biology safeguard on both attempts in each mode. Had I trusted my own flag instead of the transcript, 8 of 61 comparisons would have quietly run on a model I did not ask for.
The marking was done by a model, with me as referee. The same model marked every answer twice, the second time in a fresh session blind to the first, and neither pass saw the PDFs. Where they split, I ruled, under rules written down and applied to both modes. One rule, that a key point counts only if its details are present, is why so few answers score a full y and why I lead with the lenient score. A later blind re-mark led me to change one ruling. Agreement between two runs of one model shows consistency, not independent judgment.
So, does it still matter?
If your folder is small and you ask one question at a time, the built-in reader is enough. It gives the same answers, and on that part, the part that matters most, the harness has caught up.
An agentic RAG tool still earns its place in three ways: it answers nearly four times faster, it points to the section where the evidence sits, and its cost barely moved as the folder grew. The first two I measured. The third I measured at two folder sizes, and it is the one I expect to matter most.
The benchmark changed how I think about retrieval. At 120 papers, it did not make Claude smarter. It made getting evidence to Claude faster and cheaper. The answers were almost identical; the work and time it took to produce them were not.
The test also gave me work to do on pdf-mcp:
- The file cap. A guard I never tested beyond broke every trial on a modest folder. pdf-mcp 3.6.0 raises the cap to 500, so a folder that size gets searched in one ranking instead of split across calls.
- The 22 missed papers. I have since gone through all 22. In 13 of them, search put the paper near the top and the model picked others anyway. It set seven aside, and in the other six a near-duplicate paper got the slot. On one question the right paper ranked fourth, and the model dismissed it as “a modelling study” without opening it. Five never came up at all, and four of those have nothing to do with the question they were paired with. Only four misses came down to ranking or a poor excerpt. So the next change is to how results are presented to the model, rather than to ranking.
pdf-mcp is open source at github.com/jztan/pdf-mcp. The benchmark uses questions from the published DocHop-QA dataset; the methodology, exclusions and scoring rules are described above. If you run a similar comparison and get different numbers, I want to know.
