The first author of the Transformer paper is Ashish Vaswani. For one release, my PDF tool was certain it was Niki Parmar. The model never saw a name out of order. The extractor handed it a scrambled list before the LLM was ever involved.
That is the trap with multi-column PDFs. When a two-column paper feeds your retrieval pipeline and the answers come back garbled, it looks like a generation problem. It is a parsing problem. The lines were never in the right order to begin with.
I maintain pdf-mcp, an open-source MCP server for PDF extraction and search, with a reading-order benchmark scored against READoc ground truth on 44 arXiv papers. This is the story of fixing reading order, breaking it worse, and the metric that stayed quiet through both.
TL;DR: PDFs store text as positioned fragments with no reading order. Sorting blocks by position interleaves two columns into nonsense. Detecting columns fixes that, but naive detection reads author grids on title pages column-major and scrambles author order. Gate the column path on tall boxes that run most of the page height. Genuine columns do; grid cells don’t.
If you have never compared your extracted text to the PDF you can actually see, your pipeline is probably reading two-column papers in the wrong order right now.
The text your pipeline reads is not the text you see
Reading order is not stored in a PDF. A PDF does not store paragraphs; it stores fragments of text, each pinned to an (x, y) position on the page. The sequence those fragments come out in is largely the PDF creator’s responsibility, and many generators make little effort to preserve human reading order. The bytes arrive in whatever order the generator emitted them.
The naive fix is to sort fragments by position, top to bottom, left to right. In PyMuPDF that is one argument:
blocks = page.get_text("blocks", sort=True)
text = "\n\n".join(b[4] for b in blocks if b[6] == 0)
On a single-column page this is correct and fast. On a two-column page it is a disaster. Sorting by vertical position walks across both columns at the same height, so it reads the first line of the left column, jumps to the first line of the right column, comes back for the second line of the left, and so on.
Every sentence gets cut in half and stitched to a sentence from the other column. Once reading order is wrong, every downstream stage inherits the corruption. Chunking produces chunks that never existed in the source document. Embeddings encode mixed concepts from two unrelated columns. Your keyword and semantic search then retrieve passages that look relevant but carry half of two different arguments, and nothing in the stack tells you.
Fixing the columns: from 0.564 to 0.816
The fix is to detect column regions first, then read each one fully before moving to the next. The optional pymupdf4llm layout helper exposes column boxes for a page, so the extractor became:
boxes = detect_column_boxes(page)
if is_multi_column(boxes):
parts = (
page.get_text("text", clip=box, sort=True).strip()
for box in boxes
)
text = "\n\n".join(p for p in parts if p)
else:
# single-column path, byte-for-byte unchanged
...
To prove it worked I needed a number, not a vibe. The benchmark scores extracted text against READoc ground truth, a corpus of human-annotated markdown for real arXiv papers, using difflib.SequenceMatcher on normalized token streams: a 1.0 means identical reading order and content, lower means order scrambled or content lost. It runs on 22 two-column and 22 one-column arXiv papers.
The upper-bound column below is not 1.0, and it should not be. It is the score of PyMuPDF4LLM’s full markdown conversion, the strongest extractor in the comparison and a heavier method I deliberately do not run. Even it falls short of a perfect match, because normalization and extraction differences against the ground truth keep any extractor from scoring 1.0. So 0.860 is a practical reference ceiling, not a target.
| Group | Before | After | Upper bound |
|---|---|---|---|
| Two-column (22) | 0.564 | 0.816 | 0.860 |
| One-column (22) | 0.821 | 0.836 | 0.860 |
Two-column fidelity jumped from 0.564 to 0.816. One-column moved barely at all, which is exactly what you want: the fix should not touch pages it was never meant to change. Knowing the headroom above 0.816 matters more than closing it.
The fix that broke the title page
Here is where it got worse before it got better. The first version of the detector treated any page with more than one detected column box as multi-column. That is a reasonable assumption until you hit the first page of an academic paper.
Title pages put author names and affiliations in a grid. Three names across, affiliations underneath, sometimes more rows. The detector saw multiple boxes side by side and concluded “columns,” then read them top to bottom, column by column. A grid meant to be read across each row got read down each column instead. The author list came out reordered, and the name that should have led, Ashish Vaswani, surfaced behind Niki Parmar.
This is the distinction the first fix missed: columns versus grids. A text column and a grid cell both look like a vertical box to a naive detector. They are not the same thing, and treating them the same trades one scrambling bug for another.
The tall-box gate
The signal that separates a real column from a grid cell is height. A genuine text column runs most of the page. An author cell, a figure caption, a sidebar label occupies a short band. So the column path now requires at least two tall boxes, each at least 25% of the tallest box’s height, before it commits to column-aware reading. The 25% line is deliberately generous rather than a tuned optimum: a real column runs most of the page height while author cells and captions sit well below a quarter of it, so the threshold separates the two with room to spare.
Pages that fail the gate fall back to the plain positional sort, which is correct for the title page anyway. The author grid is short and wide, so it never trips the tall-box gate and the names come out in document order. Height turns out to be a cheap but effective signal for telling a genuine column apart from a grid cell.
When the extraction logic changed, every PDF already in the cache held text extracted by the old, broken code. A bumped extraction version (a PRAGMA user_version marker on the SQLite cache) drops the stale text, embeddings, and search index so they re-extract on the next read. Fixing the code is half the job; invalidating the poisoned cache is the other half. Otherwise retrieval keeps serving chunks and embeddings built from the old reading order, and the bug looks unfixed even after the new code ships.
The metric said the fix was clean. It wasn’t.
Here is the part that should make you nervous. When the tall-box gate shipped, the reading-order benchmark barely moved. Two-column fidelity stayed flat, one-column stayed flat. By the number, the release was a no-op.
But a real correctness bug was fixed. So why did the metric not see it?
Because one title page is a rounding error inside the benchmark. The score reads the first six pages of each paper and caps at 1500 tokens. A scrambled author block is a few dozen tokens against that total. The mean drowns it. The benchmark that caught the two-column problem, where half of every page was wrong, was structurally blind to the grid problem, where one small region of one page was wrong.
This is the aggregate blind spot: an averaged quality metric hides localized corruption, and the smaller and more important the corrupted region, the better it hides. A scrambled author list or a mangled abstract barely dents the mean, yet those are exactly the spans a retrieval query is most likely to hit.
I only trusted the aggregate number until a single wrong author name showed me what it was averaging away.
What to check in your own pipeline
If you ingest PDFs into a RAG system, the failure mode is silent by default. Extraction does not raise. Search does not warn. You get plausible chunks in the wrong order. A few concrete checks:
| Check | Why it matters |
|---|---|
| Eyeball the extracted text against the rendered page | Pick a two-column paper and a title page with an author grid. If you have never done this once, do it today. |
| Have a reading-order fidelity metric, not just “did we get text” | Token count tells you nothing about order. A sequence-similarity score against known-good text does. |
| Treat reading order as a concern separate from extraction | “We extracted the text” and “we extracted the text in the right order” are different claims with different bugs. |
| Spot-check your worst documents, not your mean | The aggregate blind spot lets a healthy average sit on top of a few badly corrupted, high-value spans. Sort by lowest per-document score and read the bottom of the list. |
The takeaway
Multi-column PDFs do not fail loudly. They hand your pipeline text that is grammatical, complete, and in the wrong order, and every layer downstream trusts it. It is one of the few bugs that poisons every stage of a RAG pipeline before retrieval: the embeddings are wrong because the reading order was wrong. Fixing the columns is the obvious win. The trap is that the fix has its own edge cases, and a single aggregate metric will happily tell you everything is fine while a paper’s first author quietly gets renamed.
Detect columns, but gate on height. Measure reading order, but never trust the mean alone. The text you can see and the text your model reads are two different documents until you prove otherwise.
