9 ms·

PDF to Text, a challenging problem

357 points by ingve 1y ago

rad_gruchalski 1y ago
So many of these problems have been solved by mozilla pdf.js together with its viewer implementation: https://mozilla.github.io/pdf.js/ https://mozilla.github.io/pdf.js/.
- zzleeper 1y ago
  Any sense on how PDF.js compares against other tools such as pdfminer?
  - rad_gruchalski 1y ago
    I don’t know. I use pdf.js for everything PDF.
  - favorited 1y ago
    I did some very broad testing of several PDF text extraction tools recently, and PDF.js was one of the slowest. My use-case was specifically testing their performance as command-line tools, so that will skew the results to an extent. For example, PDFBox was very slow because you're paying the JVM startup cost with each invocation. Poppler's pdftotext utility and pdfminer.six were generally the fastest. Both produced serviceable plain-text versions of the PDFs, with minor differences in where they placed paragraph breaks. I also wrote a small program which extracted text using Chrome's PDFium, which also performed well, but building that project can be a nightmare unless you're Google. IBM's Docling project, which uses ML models, produced by far the best formatting, preserving much of the document's original structure – but it was, of course, enormously slower and more energy-hungry. Disclaimer: I was testing specific PDF files that are representative of the kind of documents my software produces.
- egnehots 1y ago
  I don't think so, pdf.js is able to render a pdf content. Which is different from extracting "text". Text in PDF can be encoded in many ways, in an actual image, in shapes (think, segments, quadratic bezier curves...), or in an XML format (really easy to process). PDF viewers are able to render text, like a printer would work, processing command to show pixels on the screen at the end. But often, paragraph, text layout, columns, tables are lost in the process. Even though, you see them, so close yet so far. That is why AI is quite strong at this task.
  - lionkor 1y ago
    Correct me if im wrong, but pdf.js actually has a lot of methods to manipulate PDFs, no?
    - rad_gruchalski 1y ago
      Yes, pdf.js can do that: https://github.com/mozilla/pdf.js/blob/master/web/viewer.html#L284 https://github.com/mozilla/pdf.js/blob/master/web/viewer.htm.... The purpose of my original comment was to simply say: there’s an existing implementation so if you’re building a pdf file viewer/editor, and you need inspiration, have a look. One of the reasons why mozilla is doing this is to be a reference implementation. I’m not sure why people are upset with this. Though, I could have explained it better.
  - rad_gruchalski 1y ago
    You are wrong. Pdf.js can extract text and has all facilities required to render and extract formatting. The latest version can also edit PDF files. It’s basically the same engine as the Firefox PDF viewer. Which also has a document outline, search, linking, print preview, scaling, scripting sandbox… it does not simply „render” a file. Regarding tables, this here https://www.npmjs.com/package/pdf-table-extractor https://www.npmjs.com/package/pdf-table-extractor does a very good job at table interpretation and works on top of pdf.js. I also didn’t say what works better or worse, neither do I go into PDF being good or bad. I simply said that a ton of problems have been covered by
- iAMkenough 1y ago
  A good PDF reader makes the problems easier to deal with, but does not solve the underlying issue. The PDF itself is still flawed, even if pdf.js interprets it perfectly, which is still a problem for non-pdf.js viewers and tasks where "viewing" isn't the primary goal.
  - rad_gruchalski 1y ago
    Yeah. What I’m saying: pdf.js seems to have some of these solved. All I’m suggesting is have a look at it. I get it that for some PDF is a broken format.
bartread 1y ago
Yeah, getting text - even structured text - out of PDFs is no picnic. Scraping a table out of an HTML document is often straightforward even on sites that use the "everything's a <div>" (anti-)pattern, and especially on sites that use more semantically useful elements, like <table>. Not so PDFs. I'm far from an expert on the format, so maybe there is some semantic support in there, but I've seen plenty of PDFs where tables are simply an loose assemblage of graphical and text elements that, only when rendered, are easily discernible as a table because they're positioned in such a way that they render as a table. I've actually had decent luck extracting tabular data from PDFS by converting the PDFs to HTML using the Poppler PDF utils, then finding the expected table header, and then using the x-coordinate of the HTML elements for each value within the table to work out columns, and extract values for each rows. It's kind of groaty but it seems reliable for what I need. Certainly much moreso than going via formatted plaintext, which has issues with inconsistent spacing, and the insertion of newlines into the middle of rows.
- j45 1y ago
  PDFs inherently are a markup / xml format, the standard is available to learn from. It's possible to create the same PDF in many, many, many ways. Some might lean towards exporting a layout containing text and graphics from a graphics suite. Others might lean towards exporting text and graphics from a word processor, which is words first. The lens of how the creating app deals with information is often something that has input on how the PDF is output. If you're looking for an off the shelf utility that is surprisingly decent at pulling structured data from PDFs, tools like cisdem have already solved enough of it for local users. Lots of tools like this out there, many do promise structured data support but it needs to match what you're up to.
  - layer8 1y ago
    > PDFs inherently are a markup / xml format This is false. PDFs are an object graph containing imperative-style drawing instructions (among many other things). There’s a way to add structural information on top (akin to an HTML document structure), but that’s completely optional and only serves as auxiliary metadata, it’s not at the core of the PDF format.
j45 1y ago
Part of a problem being challenging is recognizing if it's new, or just new to us. We get to learn a lot when something is new to us.. at the same time the untouchable parts of PDF to Text are largely being solved with the help of LLMs. I built a tool to extract information from PDFs a long time ago, and the break through was having no ego or attachment to any one way of doing it. Different solutions and approaches offered different depth or quality of solutions and organizing them to work together in addition to anything I built myself provided what was needed - one place where more things work.. than not.
xnx 1y ago
Weird that there's no mention of LLMs in this article even though the article is very recent. LLMs haven't solved every OCR/document data extraction problem, but they've dramatically improved the situation.
- j45 1y ago
  LLMs are definitely helping approach some problems that couldn't be to date.
- simonw 1y ago
  I've had great results against PDFs from recent vision models. Gemini, OpenAI and Claude can all accept PDFs directly now and treat them as image input. For longer PDFs I've found that breaking them up into images per page and treating each page separately works well - feeing a thousand page PDF to even a long context model like Gemini 2.5 Pro or Flash still isn't reliable enough that I trust it. As always though, the big challenge of using vision LLMs for OCR (or audio transcription) tasks is the risk of accidental instruction following - even more so if there's a risk of deliberately malicious instructions in the documents you are processing.
- marginalia_nu 1y ago
  Author here: LLMs are definitely the new gold standard for smaller bodies of shorter documents. The article is in the context of an internet search engine, the corpus to be converted is of order 1 TB. Running that amount of data through an LLM would be extremely expensive, given the relatively marginal improvement in outcome.
  - mediaman 1y ago
    Corpus size doesn't mean much in the context of a PDF, given how variable that can be per page. I've found Google's Flash to cut my OCR costs by about 95+% compared to traditional commercial offerings that support structured data extraction, and I still get tables, headers, etc from each page. Still not perfect, but per page costs were less than one tenth of a cent per page, and 100 gb collections of PDFs ran to a few hundreds of dollars.
  - 1y ago
svat 1y ago
One thing I wish someone would write is something like the browser's developer tools ("inspect elements") for PDF — it would be great to be able to "view source" a PDF's content streams (the BT … ET operators that enclose text, each Tj operator for setting down text in the currently chosen font, etc), to see how every “pixel” of the PDF is being specified/generated. I know this goes against the current trend / state-of-the-art of using vision models to basically “see” the PDF like a human and “read” the text, but it would be really nice to be able to actually understand what a PDF file contains. There are a few tools that allow inspecting a PDF's contents (https://news.ycombinator.com/item?id=41379101 https://news.ycombinator.com/item?id=41379101) but they stop at the level of the PDF's objects, so entire content streams are single objects. For example, to use one of the PDFs mentioned in this post, the file https://bfi.uchicago.edu/wp-content/uploads/2022/06/BFI_WP_2022-68-1.pdf https://bfi.uchicago.edu/wp-content/uploads/2022/06/BFI_WP_2... has, corresponding to page number 6 (PDF page 8), a content stream that starts like (some newlines added by me): 0 g 0 G 0 g 0 G BT /F19 10.9091 Tf 88.936 709.041 Td [(Subsequen)28(t)-374(to)-373(the)-373(p)-28(erio)-28(d)-373(analyzed)-373(in)-374(our)-373(study)83(,)-383(Bridge's)-373(paren)27(t)-373(compan)28(y)-373(Ne)-1(wGlob)-27(e)-374(reduced)]TJ -16.936 -21.922 Td [(the)-438(n)28(um)28(b)-28(er)-437(of)-438(priv)56(ate)-438(sc)28(ho)-28(ols)-438(op)-27(erated)-438(b)28(y)-438(Bridge)-437(from)-438(405)-437(to)-438(112,)-464(and)-437(launc)28(hed)-438(a)-437(new)-438(mo)-28(del)]TJ 0 -21.923 Td and it would be really cool to be able to see the above “source” and the rendered PDF side-by-side, hover over one to see the corresponding region of the other, etc, the way we can do for a HTML page.
- whenc 1y ago
  Try with cpdf (disclaimer, wrote it): cpdf -output-json -output-json-parse-content-streams in.pdf -o out.json Then you can play around with the JSON, and turn it back to PDF with cpdf -j out.json -o out.pdf No live back-and-forth though.
  - svat 1y ago
    The live back-and-forth is the main point of what I'm asking for — I tried your cpdf (thanks for the mention; will add it to my list) and it too doesn't help; all it does is, somewhere 9000-odd lines into the JSON file, turn the part of the content stream corresponding to what I mentioned in the earlier comment into: [ [ { "F": 0.0 }, "g" ], [ { "F": 0.0 }, "G" ], [ { "F": 0.0 }, "g" ], [ { "F": 0.0 }, "G" ], [ "BT" ], [ "/F19", { "F": 10.9091 }, "Tf" ], [ { "F": 88.93600000000001 }, { "F": 709.0410000000001 }, "Td" ], [ [ "Subsequen", { "F": 28.0 }, "t", { "F": -374.0 }, "to", { "F": -373.0 }, "the", { "F": -373.0 }, "p", { "F": -28.0 }, "erio", { "F": -28.0 }, "d", { "F": -373.0 }, "analyzed", { "F": -373.0 }, "in", { "F": -374.0 }, "our", { "F": -373.0 }, "study", { "F": 83.0 }, ",", { "F": -383.0 }, "Bridge's", { "F": -373.0 }, "paren", { "F": 27.0 }, "t", { "F": -373.0 }, "compan", { "F": 28.0 }, "y", { "F": -373.0 }, "Ne", { "F": -1.0 }, "wGlob", { "F": -27.0 }, "e", { "F": -374.0 }, "reduced" ], "TJ" ], [ { "F": -16.936 }, { "F": -21.922 }, "Td" ], This is just a more verbose restatement of what's in the PDF file; the real questions I'm asking are: - How can a user get to this part, from viewing the PDF file? (Note that the PDF page objects are not necessarily a flat list; they are often nested at different levels of “kids”.) - How can a user understand these instructions, and “see” how they correspond to what is visually displayed on the PDF file?
wrs 1y ago
Since these are statistical classification problems, it seems like it would be worth trying some old-school machine learning (not an LLM, just an NN) to see how it compares with these manual heuristics.
- marginalia_nu 1y ago
  I imagine that would work pretty well given an adequate and representative body of annotated sample data. Though that is also not easy to come by.
  - ted_dunning 1y ago
    Actually, it is easy to come up with reasonably decent heuristics that can auto-tag a corpus. From that you can look for anomalies and adjust your tagging system. The problem of getting a representative body is (surprisingly) much harder than the annotation. I know. I spent quite some time years ago doing this.
  - wrs 1y ago
    But if you believe in your manual heuristics enough to ship them, you must already have a body of tests that you're happy with, right? Also seems like this is a case where generating synthetic data would be a big help. You don't have to use only real-world documents for training, just examples of the sorts of things real-world documents have in them. Make a vast corpus of semi-random documents in semi-random fonts and settings, printed from Word, Pandoc, LaTeX, etc.
andrethegiant 1y ago
Cloudflare’s ai.toMarkdown() function available in Workers AI can handle PDFs pretty easily. Judging from speed alone, it seems they’re parsing the actual content rather than shoving into OCR/LLM. Shameless plug: I use this under the hood when you prefix any PDF URL with https://pure.md/ https://pure.md/ to convert to raw text.
- burkaman 1y ago
  If you're looking for test cases, this is the first thing I tried and the result is very bad: https://pure.md/https://docs.house.gov/meetings/IF/IF00/20250513/118261/HMKP-119-IF00-20250513-SD003.pdf https://pure.md/https://docs.house.gov/meetings/IF/IF00/2025...
  - andrethegiant 1y ago
    Apart from lacking newlines, how is the result bad? It extracts the text for easy piping into an LLM.
    - burkaman 1y ago
      - Most of the titles have incorrectly split words, for example "P ART 2—R EPEAL OF EPA R ULE R ELATING TO M ULTI -P OLLUTANT E MISSION S TANDARDS". I know LLMs are resilient against typos and mistakes like this, but it still seems not ideal. - The header is parsed in a way that I suspect would mislead an LLM: "BRETT GUTHRIE, KENTUCKY FRANK PALLONE, JR., NEW JERSEY CHAIRMAN RANKING MEMBER ONE HUNDRED NINETEENTH CONGRESS". Guthrie is the chairman and Pallone is the ranking member, but that isn't implied in the text. In this particular case an LLM might already know that from other sources, but in more obscure contexts it will just have to rely on the parsed text. - It isn't converted into Markdown at all, the structure is completely lost. If you only care about text then I guess that's fine, and in this case an LLM might do an ok job at identifying some of the headers, but in the context of this discussion I think ai.toMarkdown() did a bad job of converting to Markdown and a just ok job of converting to text. I would have considered this a fairly easy test case, so it would make me hesitant to trust that function for general use if I were trying to solve the challenges described in the submitted article (Identifying headings, Joining consecutive headings, Identifying Paragraphs). I see that you are trying to minimize tokens for LLM input, so I realize your goals are probably not the same as what I'm talking about. Edit: Another test case, it seems to crash on any Arxiv PDF. Example: https://pure.md/https://arxiv.org/pdf/2411.12104 https://pure.md/https://arxiv.org/pdf/2411.12104.
bob1029 1y ago
When accommodating the general case, solving PDF-to-text is approximately equivalent to solving JPEG-to-text. The only PDF parsing scenario I would consider putting my name on is scraping AcroForm field values from standardized documents.
- kapitalx 1y ago
  This is approximately the approach we're taking also at https://doctly.ai https://doctly.ai, add to that a "multiple experts" approach for analyzing the image (for our 'ultra' version), and we get really good results. And we're making it better constantly.
- layer8 1y ago
  If you assume standardized documents, you can impose the use of Tagged PDF: https://pdfa.org/resource/tagged-pdf-q-a/ https://pdfa.org/resource/tagged-pdf-q-a/
dwheeler 1y ago
The better solution is to embed, in the PDF, the editable source document. This is easily done by LibreOffice. Embedding it takes very little space in general (because it compresses well), and then you have MUCH better information on what the text is and its meaning. It works just fine with existing PDF readers.
- layer8 1y ago
  That’s true, but it also opens up the vulnerability of the source document being arbitrarily different from the rendered PDF content.
- kerkeslager 1y ago
  That's true, but it's dependent on the creator of the PDF having aligned incentives with the consumer of the PDF. In the e-Discovery field, it's commonplace for those providing evidence to dump it into a PDF purely so that it's harder for the opposing side's lawyers to consume. If both sides have lots of money this isn't a barrier, but for example public defenders don't have funds to hire someone (me!) to process the PDFs into a readable format, so realistically they end up taking much longer to process the data, which takes a psychological toll on the defendant. And that's if they process the data at all. The solution is to make it illegal to do this: wiretap data, for example, should be provided in a standardized machine-readable format. There's no ethical reason for simple technical friction to be affecting the outcomes of criminal proceedings.
  - giovannibonetti 1y ago
    I wonder if AI will solve that
    - GaggiX 1y ago
      There are specialized models, but even generic ones like Gemini 2.0 Flash are really good and cheap, you can use them and embed the OCR inside the PDF to index to the original content.
Obscurity4340 1y ago
Is this what GoodPDF does?
reify 1y ago
https://github.com/jalan/pdftotext https://github.com/jalan/pdftotext pdftotext -layout input.pdf output.txt pip install pdftotext
EmilStenstrom 1y ago
I think using Gemma3 in vision mode could be a good use-case for converting PDF to text. It’s downloadable and runnable on a local computer, with decent memory requirements depending on which size you pick. Did anyone try it?
- CaptainFever 1y ago
  Kind of unrelated, but Gemma 3's weights are unfree, so perhaps LLaVA (https://ollama.com/library/llava https://ollama.com/library/llava) would be a good alternative.
- ljlolel 1y ago
  Mistral OCR has the best in class document understanding. https://mistral.ai/news/mistral-ocr https://mistral.ai/news/mistral-ocr
ted_dunning 1y ago
One of my favorite documents for highlighting the challenges described here is the PDF for this article: https://academic.oup.com/auk/article/126/4/717/5148354 https://academic.oup.com/auk/article/126/4/717/5148354 The first page is classic with two columns of text, centered headings, a text inclusion that sits between the columns and changes the line lengths and indentations for the columns. Then we get the fun of page headers that change between odd and even pages and section header conventions that vary drastically. Oh... to make things even better, paragraphs doing get extra spacing and don't always have an indented first line. Some of everything.
- JKCalhoun 1y ago
  The API in CoreGraphics (MacOS) for PDF, at a basic level, simply presented the text, per page, in the order in which it was encoded in the dictionaries. And 95% of the time this was pretty good — and when working with PDFKit and Preview on the Mac, we got by with it for years. If you stepped back you could imagine the app that originally had captured/produced the PDF — perhaps a word processor — it was likely rendering the text into the PDF context in some reasonable order from it's own text buffer(s). So even for two columns, you rather expect, and often found, that the text flowed correctly from the left column to the right. The text was therefore already in the correct order within the PDF document. Now, footers, headers on the page — that would be anyone's guess as to what order the PDF-producing app dumped those into the PDF context.
devrandoom 1y ago
I currently use ocrmypdf for my private library. Then Recoll to index and search. Is there a better solution I'm missing?
constantinum 1y ago
PDF parsing is hell indeed, with all sorts of edge cases that breaks business workflows, more on that here https://unstract.com/blog/pdf-hell-and-practical-rag-applications/ https://unstract.com/blog/pdf-hell-and-practical-rag-applica...
gibsonf1 1y ago
We[1] Create "Units of Thought" from PDF's and then work with those for further discovery where a "Unit of Thought" is any paragraph, title, note heading - something that stands on its own semantically. We then create a hierarchy of objects from that pdf in the database for search and conceptual search - all at scale. [1] https://graphmetrix.com/trinpod-server https://graphmetrix.com/trinpod-server https://trinapp.com https://trinapp.com
- IIAOPSW 1y ago
  I'm tempted to try it. My use case right now is a set of documents which are annual financial and statutory disclosures of a large institution. Every year they are formatted / organized slightly differently which makes it enormously tedious to manually find and compare the same basic section from one year to another, but they are consistent enough to recognize analogous sections from different years due to often reusing verbatim quotes or highly specific key words each time. What I really want to do is take all these docs and just reorder all the content such that I can look at page n (or section whatever) scrolling down and compare it between different years by scrolling horizontally. Ideally with changes from one year to the next highlighted. Can your product do this?
  - gibsonf1 1y ago
    Probably without too much difficulty. If you have a sample to confirm, that would be great. frederick @ graphmetrix . com
kbyatnal 1y ago
"PDF to Text" is a bit simplified IMO. There's actually a few class of problems within this category: 1. reliable OCR from documents (to index for search, feed into a vector DB, etc) 2. structured data extraction (pull out targeted values) 3. end-to-end document pipelines (e.g. automate mortgage applications) Marginalia needs to solve problem #1 (OCR), which is luckily getting commoditized by the day thanks to models like Gemini Flash. I've now seen multiple companies replace their OCR pipelines with Flash for a fraction of the cost of previous solutions, it's really quite remarkable. Problems #2 and #3 are much more tricky. There's still a large gap for businesses in going from raw OCR outputs —> document pipelines deployed in prod for mission-critical use cases. LLMs and VLMs aren't magic, and anyone who goes in expecting 100% automation is in for a surprise. You still need to build and label datasets, orchestrate pipelines (classify -> split -> extract), detect uncertainty and correct with human-in-the-loop, fine-tune, and a lot more. You can certainly get close to full automation over time, but it's going to take time and effort. The future is definitely moving in this direction though. Disclaimer: I started a LLM doc processing company to help companies solve problems in this space (https://extend.ai https://extend.ai)
- varunneal 1y ago
  I've been hacking away at trying to process PDFs into Markdown, having encountered similar obstacles to OP regarding header detection (and many other issues). OCR is fantastic these days but maintaining a global structure to the document is much trickier. Consistent HTML seems still out of reach for large documents. I'm having half-decent results with Markdown using multiple passes of an LLM to extract document structure and feeding it in contextually for page-by-pass extraction.
  - dstryr 1y ago
    Give this project a try. I've been using it with promising results. https://github.com/matthsena/AlcheMark https://github.com/matthsena/AlcheMark
    - aorth 1y ago
      I tried with one PDF and was surprised to see it connect to some cloud service: 2025-05-14 07:58:49,373 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): openaipublic.blob.core.windows.net:443 2025-05-14 07:58:50,446 - urllib3.connectionpool - DEBUG - https://openaipublic.blob.core.windows.net:443 "GET /encodings/o200k_base.tiktoken HTTP/1.1" 200 361 3922 The project's README doesn't mention that anywhere...
anonu 1y ago
They should called it NDF - Non-Portable Document Format.
- 0xml 1y ago
  Maybe Printable Document Format.
dobraczekolada 1y ago
Reminds me of github.com/docwire/docwire
90s_dev 1y ago
Have any of you ever thought to yourself, this is new and interesting, and then vaguely remembered that you spent months or years becoming an expert at it earlier in life but entirely forgot it? And in fact large chunks of the very interesting things you've done just completely flew out of your mind long ago, to the point where you feel absolutely new at life, like you've accomplished relatively nothing, until something like this jars you out of that forgetfulness? I definitely vaguely remember doing some incredibly cool things with PDFs and OCR about 6 or 7 years ago. Some project comes to mind... google tells me it was "tesseract" and that sounds familiar.
- deleted 1y ago
  [deleted]
- downboots 1y ago
  No different than a fire ant whose leaf got knocked over by the wind and it moved on to the next.
  - 90s_dev 1y ago
    Well I sure do feel different than a fire ant.
    - downboots 1y ago
      anttention is all we have
      - 90s_dev 1y ago
        Not true, I also have a nice cigar waiting for the rain to go away.
        90s_dev 1y ago
        Hmm, it's gone now. Well I used to have one anyway.
- 1y ago
PeterStuer 1y ago
I guess I'm lucky the PDF's I need to process are mostly rather dull unadventurous layouts. So far I've had great success using docling.
keybored 1y ago
For people who want people to read their documents[1] they should have their PDF point to a more digital-friendly format, an alt document. Looks like you’ve found my PDF. You might want this version instead: PDFs are often subpar. Just see the first example: standard Latex serif section title. I mean, PDFs often aren’t even well-typeset for what they are (dead-tree simulations). [1] No sarcasm or truism. Some may just want to submit a paper to whatever publisher and go through their whole laundry list of what a paper ought to be. Wide dissemanation is not the point.
1vuio0pswjnm7 1y ago
Below is a PDF. It is a .txt file. I can save it with a .pdf extension and open it in a PDF viewer. I can make changes in a text editor. For example, by editing this text file, I can change the text displayed on the screen when the PDF is opened, the font, font size, line spacing, the maximum characters per line, number of lines per page, the paper width and height, as well as portrait versus landscape mode. %PDF-1.4 1 0 obj << /CreationDate (D:2025) /Producer >> endobj 2 0 obj << /Type /Catalog /Pages 3 0 R >> endobj 4 0 obj << /Type /Font /Subtype /Type1 /Name /F1 /BaseFont /Times-Roman >> endobj 5 0 obj << /Font << /F1 4 0 R >> /ProcSet [ /PDF /Text ] >> endobj 6 0 obj << /Type /Page /Parent 3 0 R /Resources 5 0 R /Contents 7 0 R >> endobj 7 0 obj << /Length 8 0 R >> stream BT /F1 50 Tf 1 0 0 1 50 752 Tm 54 TL (PDF is)' ((a) a text format)' ((b) a graphics format)' ((c) (a) and (b).)' ()' ET endstream endobj 8 0 obj 53 endobj 3 0 obj << /Type /Pages /Count 1 /MediaBox [ 0 0 612 792 ] /Kids [ 6 0 R ] >> endobj xref 0 9 0000000000 65535 f 0000000009 00000 n 0000000113 00000 n 0000000514 00000 n 0000000162 00000 n 0000000240 00000 n 0000000311 00000 n 0000000391 00000 n 0000000496 00000 n trailer << /Size 9 /Root 2 0 R /Info 1 0 R >> startxref 599 %%EOF
- swsieber 1y ago
  It can also have embedded binary streams. It was not made for text. It was made for layout and graphics. You give nice examples, but each of those lines could have been broken up into one call per character, or per word, even out of order.
  - hnick 1y ago
    It can also use fonts which map glyphs via characters which do not represent the final visual item e.g. "PDF" could be "1#F" and you only really know what it looks like by rendering then viewing/OCR. A nice file won't, but sometimes the best work is in not dealing with nice things.
    - 90s_dev 1y ago
ljlolel 1y ago
Mistral OCR has best in class doing document understanding https://mistral.ai/news/mistral-ocr https://mistral.ai/news/mistral-ocr
nicodjimenez 1y ago
Check out mathpix.com. We handle complex tables, complex math, diagrams, rotated tables, and much more, extremely accurately. Disclaimer: I'm the founder.
bickfordb 1y ago
Maybe it's time for new document formats and browsers that neatly separate content, presentation and UI layers? PDF and HTML are 20+ years old and it's often difficult to extract information from either let alone author a browser.
- rrr_oh_man 1y ago
  Yes, but I'm sure they're out there somewhere (https://xkcd.com/927/ https://xkcd.com/927/)
  - TRiG_Ireland 1y ago
    Open XML Paper Specification is an XML-based format intended to compete with PDF. Unlike PDF, it is purely static: no scripting. Also unlike PDF, I've never seen it actually used in the wild.
smcleod 1y ago
Definitely recommend docling for this. https://docling-project.github.io/docling/ https://docling-project.github.io/docling/
elpalek 1y ago
Recently tested a (non-english) pdf ocr with Gemini 2.5 Pro. First, directly ask it to extract text from pdf. Result: random text blob, not useable. Second, I converted pdf into pages of jpg. Gemini performed exceptional. Near perfect text extraction with intact format in markdown. Maybe there's internal difference when processing pdf vs jpg inside the model.
- jagged-chisel 1y ago
  Model isn’t rendering the PDF probably, just looking in the file for text.
noosphr 1y ago
I've worked on this in my day job: extracting _all_ relevant information from a financial services PDF for a bert based search engine. The only way to solve that is with a segmentation model followed by a regular OCR model and whatever other specialized models you need to extract other types of data. VLM aren't ready for prime time and won't be for a decade on more. What worked was using doclaynet trained YOLO models to get the areas of the document that were text, images, tables or formulas: https://github.com/DS4SD/DocLayNet https://github.com/DS4SD/DocLayNet if you don't care about anything but text you can feed the results into tesseract directly (but for the love of god read the manual). Congratulations, you're done. Here's some pre-trained models that work OK out of the box: https://github.com/ppaanngggg/yolo-doclaynet https://github.com/ppaanngggg/yolo-doclaynet I found that we needed to increase the resolution from ~700px to ~2100px horizontal for financial data segmentation. VLMs on the other hand still choke on long text and hallucinate unpredictably. Worse they can't understand nested data. If you give _any_ current model nothing harder than three nested rectangles with text under each they will not extract the text correctly. Given that nested rectangles describes every table no VLM can currently extract data from anything but the most straightforward of tables. But it will happily lie to you that it did - after all a mining company should own a dozen bulldozers right? And if they each cost $35.000 it must be an amazing deal they got, right?
- yfontana 1y ago
  I've been working on extracting text from some 20 million PDFs, with just about every type of layout you can imagine. We're using a similar approach (segmentation / OCR), but with PyMuPDF. The full extract is projected to run for several days on a GPU cluster, at a cost of like 20-30k (can't remember the exact number but it's in that ballpark). When you can afford this kind of compute, text extraction from PDFs isn't quite a fully solved problem, but we're most of the way there. What the article in the OP tries to do is, as far as I understand, somewhat different. It's trying to use much simpler heuristics to get acceptable results cheaper and faster, and this is definitely an open issue.
- cess11 1y ago
  That looks like a pretty good starting point, thanks. I've been dabbling in vision models but need a much higher degree of accuracy than they seem able to provide, opting instead for more traditional techniques and handling errors manually.
Sharlin 1y ago
Some of the unsung heroes of the modern age are the programmers who, through what must have involved a lot of weeping and gnashing of teeth, have managed to implement the find, select, and copy operations in PDF readers.
patrick41638265 1y ago
Good old https://linux.die.net/man/1/pdftotext https://linux.die.net/man/1/pdftotext and a little Python on top of its output will get you a long way if your documents are not too crazy. I use it to parse all my bank statements into an sqlite database for analysis.
coolcase 1y ago
Tried extracting data from a newspaper. It is really hard. What is a headline and which headline belongs to which paragraphs? Harder than you think! And chucking it as is into OpenAI was no good at all. Manually dealing with coordinates from OCR was better but not perfect.
rekoros 1y ago
I've been using Azure's "Document Intelligence" thingy (prebuilt "read" model) to extract text from PDFs with pretty good results [1]. Their terminology is so bad, it's easy to dismiss the whole thing for another Microsoft pile, but it actually, like, for real, works. [1] https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/prebuilt/read?view=doc-intel-4.0.0&tabs=sample-code https://learn.microsoft.com/en-us/azure/ai-services/document...
incanus77 1y ago
I did some contract work some years back with a company who had a desktop product (for Mac) that would apply some smarts to strip out extraneous things on pages while printing (such as ads on webpages) as well as try to avoid the case where only a line or two was printed on a page, wasting paper. It initially was getting into things at the PostScript layer, which unsurprisingly was horrifying, but eventually worked on PDFs. This required finding and interpreting various textual parts of the passed documents and was a pretty big technical challenge. While I'm not convinced it was viable at the business level, it feels like something platform/OS companies could focus on to have a measurable environmental and cost overhead impact.
viking2917 1y ago
coincidentally, posted this over on Show HN today. OCR workbench, AI OCR & editing tools for OCRing old / hard documents. https://news.ycombinator.com/item?id=43976450 https://news.ycombinator.com/item?id=43976450. Tesseract works fine for modern text documents, but it fails badly on older docs (e.g. colonial american, etc)
gerdesj 1y ago
PDF is a display format. It is optimised for eyeballs and printers. There has been some feature creep. It is a rubbish machine data transfer mechanism but really good for humans and say storing a page of A4 (letter for the US). So, you start off with the premise that a .pdf stores text and you want that text. Well that's nice: grow some eyes! Otherwise, you are going to have to get to grips with some really complicated stuff. For starters, is the text ... text or is it an image? Your eyes don't care and will just work (especially when you pop your specs back on) but your parser is probably seg faulting madly. It just gets worse. PDF is for humans to read. Emulate a human to read a PDF.
- amai 1y ago
  That's it. I could not have said it better.
TZubiri 1y ago
As someone who has worked on this FT. (S&P, parsing of financial disclosures) The solution is OCR. Don't fuck with internal file format. PDF is designed to print/display stuff, not to be parseable by machines.
fracus 1y ago
Why hasn't the PDF standard been replaced or revised to require the text in meta form? Seems like a no brainer.
- msephton 1y ago
  It's already supported, but it's an optional feature, so it's up to the app/developer/author.
remram 1y ago
I built a simple OSS tool for qualitative data analysis, which needs to turn uploaded documents into text (stripped HTML). PDFs have been a huge problem from day one. I have investigated many tools, but two-column layouts and footers etc often still mess up the content. It's hard to convince my (often non-technical) users that this is a difficult problem.
- bartread 1y ago
  Try Poppler’s pdftohtml command line tool. For me that seems to do a good job of spitting out multi-column text in the right order. Then you have the much easier task of extracting the text from the HTML. Also, if it does come out in the wrong order for any pages you can analyse element coordinates to figure out which column each chunk of text belongs in. (Note that you may have to deal with sub-columns if tables are present in any columns. I’ve never had this in my data but you may also find blocks that span across more than one column, either in whole or in part.) They also have a pdftotext tool that may do the job for you if you disable its layout option. If you run it with the layout option enabled you’ll find it generates multi-column text in the output, as it tries to closely match the layout of the input PDF. I think the pdftohtml tool is probably the way to go just because the extra metadata on each element is probably going to be helpful in determining how to treat that element, and it’s obviously relatively straightforward to strip out the HTML tags to extract plain text.
trevor-e 1y ago
Having built some toy parsers for PDF files in the past it was a huge wtf moment for me when I realized how the format works. With that in mind, it's even more puzzling how it's used often in text-heavy cases. I always think about the invoicing use-case: digital systems should be able to easy extract data from the file while still being formatted visually for humans. It seems like the tech world would be much better off if we migrated to a better format.
- Uvix 1y ago
  XML+XSLT was almost this, but unfortunately browsers no longer support it for local XML files, only ones on a remote server.
Klaus_ 1y ago
[dead]
herodotus 1y ago
This is mostly what I worked on for many years at Apple with reasonable success. The main secret was to accept that everything was geometry, and use cluster analysis to try to distinguish between word gaps and letter gaps. On many PDF documents, it works really well, but there are so many different kinds of PDF documents that there are always cases were the results are not that great. If I were to do it today, I would stick with geometry, avoid OCR completely, but use machine learning. One big advantage for machine learning is that I could use existing tools to generate PDFs from known text, so that the training phase could be completly automatic. (Here is Bertrand Serlet announcing the feature at WWDC in 2009: https://youtu.be/FTfChHwGFf0?si=wNCfI9wZj1aj9rY7&t=308 https://youtu.be/FTfChHwGFf0?si=wNCfI9wZj1aj9rY7&t=308)
lewtun 1y ago
> The absolute best way of doing this is these days is likely through a vision based machine learning model, but that is an approach that is very far away from scaling to processing hundreds of gigabytes of PDF files off a single server with no GPU. SmolDocling is pretty fast and the ONNX weights can be scaled to many CPUs: https://huggingface.co/ds4sd/SmolDocling-256M-preview https://huggingface.co/ds4sd/SmolDocling-256M-preview Not sure what time scale the author had in mind for processing GBs of PDFs, but the future might be closer than “very far away”
snehanairdoc 1y ago
This was a great read. You've done an excellent job breaking down what makes PDFs so uniquely annoying to work with. People often underestimate how much of the “document-ness” (like headings, paragraphs, tables) is just visual, with no underlying semantic structure. We ran into many of the same challenges while working on Docsumo, where we process business documents like invoices, bank statements, and scanned PDFs. In real-world use cases, things get even messier: inconsistent templates, rotated scans, overlapping text, or documents generated by ancient software with no tagging at all. One thing we’ve found helpful (in addition to heuristics like font size/weight and spacing) is combining layout parsing with ML models trained to infer semantic roles (like "header", "table cell", "footer", etc.). It’s far from perfect, but it helps bridge the gap between how the document looks and what it means. Really appreciate posts like this. PDF wrangling is a dark art more people should talk about.
hilbert42 1y ago
"It doesn’t have text in the way you might think of it, but more of a mapping of glyphs to coordinates on “paper”." I've often had trouble extracting text from PDFs, it's time consuming and messy, so a quick question. The PDF format is now pretty ancient
hilbert42 1y ago
"It doesn’t have text in the way you might think of it, but more of a mapping of glyphs to coordinates on “paper”." I've often had trouble extracting text from PDFs, it's time consuming and messy, so a quick question. The PDF format works pretty well for what it does but it's now pretty ancient, so does anyone know if there's any newer format on the horizon that could be a next-generation replacement that would make it much easier to extract its data and export it to another format (say, docx, odt, etc.)?
drguthals 1y ago
PDF -> Useful Information is what Tensorlake does (https://tensorlake.ai https://tensorlake.ai) Because PDFs are so dominate and yet each one has information in more than just text (tables, images, formulas, hand-writing, strike-throughs even), we (as devs) need to be able have tools that understand the contents, not just "read" them. Full disclosure...I work there
nemosaltat 1y ago
`uv run marker_single “path/to/file.pdf”` auto dumps markdown, images, and a json map of the PDF to a folder in the venv and gives you the path. It’s like magic.

357 points by ingve 1y ago

So many of these problems have been solved by mozilla pdf.js together with its viewer implementation: https://mozilla.github.io/pdf.js/ https://mozilla.github.io/pdf.js/.

zzleeper 1y ago
Any sense on how PDF.js compares against other tools such as pdfminer?
- rad_gruchalski 1y ago
  I don’t know. I use pdf.js for everything PDF.
- favorited 1y ago
  I did some very broad testing of several PDF text extraction tools recently, and PDF.js was one of the slowest. My use-case was specifically testing their performance as command-line tools, so that will skew the results to an extent. For example, PDFBox was very slow because you're paying the JVM startup cost with each invocation. Poppler's pdftotext utility and pdfminer.six were generally the fastest. Both produced serviceable plain-text versions of the PDFs, with minor differences in where they placed paragraph breaks. I also wrote a small program which extracted text using Chrome's PDFium, which also performed well, but building that project can be a nightmare unless you're Google. IBM's Docling project, which uses ML models, produced by far the best formatting, preserving much of the document's original structure – but it was, of course, enormously slower and more energy-hungry. Disclaimer: I was testing specific PDF files that are representative of the kind of documents my software produces.
egnehots 1y ago
I don't think so, pdf.js is able to render a pdf content. Which is different from extracting "text". Text in PDF can be encoded in many ways, in an actual image, in shapes (think, segments, quadratic bezier curves...), or in an XML format (really easy to process). PDF viewers are able to render text, like a printer would work, processing command to show pixels on the screen at the end. But often, paragraph, text layout, columns, tables are lost in the process. Even though, you see them, so close yet so far. That is why AI is quite strong at this task.
- lionkor 1y ago
  Correct me if im wrong, but pdf.js actually has a lot of methods to manipulate PDFs, no?
  - rad_gruchalski 1y ago
    Yes, pdf.js can do that: https://github.com/mozilla/pdf.js/blob/master/web/viewer.html#L284 https://github.com/mozilla/pdf.js/blob/master/web/viewer.htm.... The purpose of my original comment was to simply say: there’s an existing implementation so if you’re building a pdf file viewer/editor, and you need inspiration, have a look. One of the reasons why mozilla is doing this is to be a reference implementation. I’m not sure why people are upset with this. Though, I could have explained it better.
- rad_gruchalski 1y ago
  You are wrong. Pdf.js can extract text and has all facilities required to render and extract formatting. The latest version can also edit PDF files. It’s basically the same engine as the Firefox PDF viewer. Which also has a document outline, search, linking, print preview, scaling, scripting sandbox… it does not simply „render” a file. Regarding tables, this here https://www.npmjs.com/package/pdf-table-extractor https://www.npmjs.com/package/pdf-table-extractor does a very good job at table interpretation and works on top of pdf.js. I also didn’t say what works better or worse, neither do I go into PDF being good or bad. I simply said that a ton of problems have been covered by
iAMkenough 1y ago
A good PDF reader makes the problems easier to deal with, but does not solve the underlying issue. The PDF itself is still flawed, even if pdf.js interprets it perfectly, which is still a problem for non-pdf.js viewers and tasks where "viewing" isn't the primary goal.
- rad_gruchalski 1y ago
  Yeah. What I’m saying: pdf.js seems to have some of these solved. All I’m suggesting is have a look at it. I get it that for some PDF is a broken format.

bartread 1y ago

Yeah, getting text - even structured text - out of PDFs is no picnic. Scraping a table out of an HTML document is often straightforward even on sites that use the "everything's a <div>" (anti-)pattern, and especially on sites that use more semantically useful elements, like <table>. Not so PDFs. I'm far from an expert on the format, so maybe there is some semantic support in there, but I've seen plenty of PDFs where tables are simply an loose assemblage of graphical and text elements that, only when rendered, are easily discernible as a table because they're positioned in such a way that they render as a table. I've actually had decent luck extracting tabular data from PDFS by converting the PDFs to HTML using the Poppler PDF utils, then finding the expected table header, and then using the x-coordinate of the HTML elements for each value within the table to work out columns, and extract values for each rows. It's kind of groaty but it seems reliable for what I need. Certainly much moreso than going via formatted plaintext, which has issues with inconsistent spacing, and the insertion of newlines into the middle of rows.

j45 1y ago
PDFs inherently are a markup / xml format, the standard is available to learn from. It's possible to create the same PDF in many, many, many ways. Some might lean towards exporting a layout containing text and graphics from a graphics suite. Others might lean towards exporting text and graphics from a word processor, which is words first. The lens of how the creating app deals with information is often something that has input on how the PDF is output. If you're looking for an off the shelf utility that is surprisingly decent at pulling structured data from PDFs, tools like cisdem have already solved enough of it for local users. Lots of tools like this out there, many do promise structured data support but it needs to match what you're up to.
- layer8 1y ago
  > PDFs inherently are a markup / xml format This is false. PDFs are an object graph containing imperative-style drawing instructions (among many other things). There’s a way to add structural information on top (akin to an HTML document structure), but that’s completely optional and only serves as auxiliary metadata, it’s not at the core of the PDF format.

j45 1y ago

Part of a problem being challenging is recognizing if it's new, or just new to us. We get to learn a lot when something is new to us.. at the same time the untouchable parts of PDF to Text are largely being solved with the help of LLMs. I built a tool to extract information from PDFs a long time ago, and the break through was having no ego or attachment to any one way of doing it. Different solutions and approaches offered different depth or quality of solutions and organizing them to work together in addition to anything I built myself provided what was needed - one place where more things work.. than not.

xnx 1y ago

Weird that there's no mention of LLMs in this article even though the article is very recent. LLMs haven't solved every OCR/document data extraction problem, but they've dramatically improved the situation.

j45 1y ago
LLMs are definitely helping approach some problems that couldn't be to date.
simonw 1y ago
I've had great results against PDFs from recent vision models. Gemini, OpenAI and Claude can all accept PDFs directly now and treat them as image input. For longer PDFs I've found that breaking them up into images per page and treating each page separately works well - feeing a thousand page PDF to even a long context model like Gemini 2.5 Pro or Flash still isn't reliable enough that I trust it. As always though, the big challenge of using vision LLMs for OCR (or audio transcription) tasks is the risk of accidental instruction following - even more so if there's a risk of deliberately malicious instructions in the documents you are processing.
marginalia_nu 1y ago
Author here: LLMs are definitely the new gold standard for smaller bodies of shorter documents. The article is in the context of an internet search engine, the corpus to be converted is of order 1 TB. Running that amount of data through an LLM would be extremely expensive, given the relatively marginal improvement in outcome.
- mediaman 1y ago
  Corpus size doesn't mean much in the context of a PDF, given how variable that can be per page. I've found Google's Flash to cut my OCR costs by about 95+% compared to traditional commercial offerings that support structured data extraction, and I still get tables, headers, etc from each page. Still not perfect, but per page costs were less than one tenth of a cent per page, and 100 gb collections of PDFs ran to a few hundreds of dollars.
- 1y ago

svat 1y ago

One thing I wish someone would write is something like the browser's developer tools ("inspect elements") for PDF — it would be great to be able to "view source" a PDF's content streams (the BT … ET operators that enclose text, each Tj operator for setting down text in the currently chosen font, etc), to see how every “pixel” of the PDF is being specified/generated. I know this goes against the current trend / state-of-the-art of using vision models to basically “see” the PDF like a human and “read” the text, but it would be really nice to be able to actually understand what a PDF file contains. There are a few tools that allow inspecting a PDF's contents (https://news.ycombinator.com/item?id=41379101 https://news.ycombinator.com/item?id=41379101) but they stop at the level of the PDF's objects, so entire content streams are single objects. For example, to use one of the PDFs mentioned in this post, the file https://bfi.uchicago.edu/wp-content/uploads/2022/06/BFI_WP_2022-68-1.pdf https://bfi.uchicago.edu/wp-content/uploads/2022/06/BFI_WP_2... has, corresponding to page number 6 (PDF page 8), a content stream that starts like (some newlines added by me): 0 g 0 G 0 g 0 G BT /F19 10.9091 Tf 88.936 709.041 Td [(Subsequen)28(t)-374(to)-373(the)-373(p)-28(erio)-28(d)-373(analyzed)-373(in)-374(our)-373(study)83(,)-383(Bridge's)-373(paren)27(t)-373(compan)28(y)-373(Ne)-1(wGlob)-27(e)-374(reduced)]TJ -16.936 -21.922 Td [(the)-438(n)28(um)28(b)-28(er)-437(of)-438(priv)56(ate)-438(sc)28(ho)-28(ols)-438(op)-27(erated)-438(b)28(y)-438(Bridge)-437(from)-438(405)-437(to)-438(112,)-464(and)-437(launc)28(hed)-438(a)-437(new)-438(mo)-28(del)]TJ 0 -21.923 Td and it would be really cool to be able to see the above “source” and the rendered PDF side-by-side, hover over one to see the corresponding region of the other, etc, the way we can do for a HTML page.

whenc 1y ago
Try with cpdf (disclaimer, wrote it): cpdf -output-json -output-json-parse-content-streams in.pdf -o out.json Then you can play around with the JSON, and turn it back to PDF with cpdf -j out.json -o out.pdf No live back-and-forth though.
- svat 1y ago
  The live back-and-forth is the main point of what I'm asking for — I tried your cpdf (thanks for the mention; will add it to my list) and it too doesn't help; all it does is, somewhere 9000-odd lines into the JSON file, turn the part of the content stream corresponding to what I mentioned in the earlier comment into: [ [ { "F": 0.0 }, "g" ], [ { "F": 0.0 }, "G" ], [ { "F": 0.0 }, "g" ], [ { "F": 0.0 }, "G" ], [ "BT" ], [ "/F19", { "F": 10.9091 }, "Tf" ], [ { "F": 88.93600000000001 }, { "F": 709.0410000000001 }, "Td" ], [ [ "Subsequen", { "F": 28.0 }, "t", { "F": -374.0 }, "to", { "F": -373.0 }, "the", { "F": -373.0 }, "p", { "F": -28.0 }, "erio", { "F": -28.0 }, "d", { "F": -373.0 }, "analyzed", { "F": -373.0 }, "in", { "F": -374.0 }, "our", { "F": -373.0 }, "study", { "F": 83.0 }, ",", { "F": -383.0 }, "Bridge's", { "F": -373.0 }, "paren", { "F": 27.0 }, "t", { "F": -373.0 }, "compan", { "F": 28.0 }, "y", { "F": -373.0 }, "Ne", { "F": -1.0 }, "wGlob", { "F": -27.0 }, "e", { "F": -374.0 }, "reduced" ], "TJ" ], [ { "F": -16.936 }, { "F": -21.922 }, "Td" ], This is just a more verbose restatement of what's in the PDF file; the real questions I'm asking are: - How can a user get to this part, from viewing the PDF file? (Note that the PDF page objects are not necessarily a flat list; they are often nested at different levels of “kids”.) - How can a user understand these instructions, and “see” how they correspond to what is visually displayed on the PDF file?

wrs 1y ago

Since these are statistical classification problems, it seems like it would be worth trying some old-school machine learning (not an LLM, just an NN) to see how it compares with these manual heuristics.

marginalia_nu 1y ago
I imagine that would work pretty well given an adequate and representative body of annotated sample data. Though that is also not easy to come by.
- ted_dunning 1y ago
  Actually, it is easy to come up with reasonably decent heuristics that can auto-tag a corpus. From that you can look for anomalies and adjust your tagging system. The problem of getting a representative body is (surprisingly) much harder than the annotation. I know. I spent quite some time years ago doing this.
- wrs 1y ago
  But if you believe in your manual heuristics enough to ship them, you must already have a body of tests that you're happy with, right? Also seems like this is a case where generating synthetic data would be a big help. You don't have to use only real-world documents for training, just examples of the sorts of things real-world documents have in them. Make a vast corpus of semi-random documents in semi-random fonts and settings, printed from Word, Pandoc, LaTeX, etc.

andrethegiant 1y ago

Cloudflare’s ai.toMarkdown() function available in Workers AI can handle PDFs pretty easily. Judging from speed alone, it seems they’re parsing the actual content rather than shoving into OCR/LLM. Shameless plug: I use this under the hood when you prefix any PDF URL with https://pure.md/ https://pure.md/ to convert to raw text.

burkaman 1y ago
If you're looking for test cases, this is the first thing I tried and the result is very bad: https://pure.md/https://docs.house.gov/meetings/IF/IF00/20250513/118261/HMKP-119-IF00-20250513-SD003.pdf https://pure.md/https://docs.house.gov/meetings/IF/IF00/2025...
- andrethegiant 1y ago
  Apart from lacking newlines, how is the result bad? It extracts the text for easy piping into an LLM.
  - burkaman 1y ago
    - Most of the titles have incorrectly split words, for example "P ART 2—R EPEAL OF EPA R ULE R ELATING TO M ULTI -P OLLUTANT E MISSION S TANDARDS". I know LLMs are resilient against typos and mistakes like this, but it still seems not ideal. - The header is parsed in a way that I suspect would mislead an LLM: "BRETT GUTHRIE, KENTUCKY FRANK PALLONE, JR., NEW JERSEY CHAIRMAN RANKING MEMBER ONE HUNDRED NINETEENTH CONGRESS". Guthrie is the chairman and Pallone is the ranking member, but that isn't implied in the text. In this particular case an LLM might already know that from other sources, but in more obscure contexts it will just have to rely on the parsed text. - It isn't converted into Markdown at all, the structure is completely lost. If you only care about text then I guess that's fine, and in this case an LLM might do an ok job at identifying some of the headers, but in the context of this discussion I think ai.toMarkdown() did a bad job of converting to Markdown and a just ok job of converting to text. I would have considered this a fairly easy test case, so it would make me hesitant to trust that function for general use if I were trying to solve the challenges described in the submitted article (Identifying headings, Joining consecutive headings, Identifying Paragraphs). I see that you are trying to minimize tokens for LLM input, so I realize your goals are probably not the same as what I'm talking about. Edit: Another test case, it seems to crash on any Arxiv PDF. Example: https://pure.md/https://arxiv.org/pdf/2411.12104 https://pure.md/https://arxiv.org/pdf/2411.12104.

bob1029 1y ago

When accommodating the general case, solving PDF-to-text is approximately equivalent to solving JPEG-to-text. The only PDF parsing scenario I would consider putting my name on is scraping AcroForm field values from standardized documents.

kapitalx 1y ago
This is approximately the approach we're taking also at https://doctly.ai https://doctly.ai, add to that a "multiple experts" approach for analyzing the image (for our 'ultra' version), and we get really good results. And we're making it better constantly.
layer8 1y ago
If you assume standardized documents, you can impose the use of Tagged PDF: https://pdfa.org/resource/tagged-pdf-q-a/ https://pdfa.org/resource/tagged-pdf-q-a/

dwheeler 1y ago

The better solution is to embed, in the PDF, the editable source document. This is easily done by LibreOffice. Embedding it takes very little space in general (because it compresses well), and then you have MUCH better information on what the text is and its meaning. It works just fine with existing PDF readers.

layer8 1y ago
That’s true, but it also opens up the vulnerability of the source document being arbitrarily different from the rendered PDF content.
kerkeslager 1y ago
That's true, but it's dependent on the creator of the PDF having aligned incentives with the consumer of the PDF. In the e-Discovery field, it's commonplace for those providing evidence to dump it into a PDF purely so that it's harder for the opposing side's lawyers to consume. If both sides have lots of money this isn't a barrier, but for example public defenders don't have funds to hire someone (me!) to process the PDFs into a readable format, so realistically they end up taking much longer to process the data, which takes a psychological toll on the defendant. And that's if they process the data at all. The solution is to make it illegal to do this: wiretap data, for example, should be provided in a standardized machine-readable format. There's no ethical reason for simple technical friction to be affecting the outcomes of criminal proceedings.
- giovannibonetti 1y ago
  I wonder if AI will solve that
  - GaggiX 1y ago
    There are specialized models, but even generic ones like Gemini 2.0 Flash are really good and cheap, you can use them and embed the OCR inside the PDF to index to the original content.

Obscurity4340 1y ago

Is this what GoodPDF does?

reify 1y ago

https://github.com/jalan/pdftotext https://github.com/jalan/pdftotext pdftotext -layout input.pdf output.txt pip install pdftotext

EmilStenstrom 1y ago

I think using Gemma3 in vision mode could be a good use-case for converting PDF to text. It’s downloadable and runnable on a local computer, with decent memory requirements depending on which size you pick. Did anyone try it?

CaptainFever 1y ago
Kind of unrelated, but Gemma 3's weights are unfree, so perhaps LLaVA (https://ollama.com/library/llava https://ollama.com/library/llava) would be a good alternative.
ljlolel 1y ago
Mistral OCR has the best in class document understanding. https://mistral.ai/news/mistral-ocr https://mistral.ai/news/mistral-ocr

ted_dunning 1y ago

One of my favorite documents for highlighting the challenges described here is the PDF for this article: https://academic.oup.com/auk/article/126/4/717/5148354 https://academic.oup.com/auk/article/126/4/717/5148354 The first page is classic with two columns of text, centered headings, a text inclusion that sits between the columns and changes the line lengths and indentations for the columns. Then we get the fun of page headers that change between odd and even pages and section header conventions that vary drastically. Oh... to make things even better, paragraphs doing get extra spacing and don't always have an indented first line. Some of everything.

JKCalhoun 1y ago
The API in CoreGraphics (MacOS) for PDF, at a basic level, simply presented the text, per page, in the order in which it was encoded in the dictionaries. And 95% of the time this was pretty good — and when working with PDFKit and Preview on the Mac, we got by with it for years. If you stepped back you could imagine the app that originally had captured/produced the PDF — perhaps a word processor — it was likely rendering the text into the PDF context in some reasonable order from it's own text buffer(s). So even for two columns, you rather expect, and often found, that the text flowed correctly from the left column to the right. The text was therefore already in the correct order within the PDF document. Now, footers, headers on the page — that would be anyone's guess as to what order the PDF-producing app dumped those into the PDF context.

devrandoom 1y ago

I currently use ocrmypdf for my private library. Then Recoll to index and search. Is there a better solution I'm missing?

constantinum 1y ago

PDF parsing is hell indeed, with all sorts of edge cases that breaks business workflows, more on that here https://unstract.com/blog/pdf-hell-and-practical-rag-applications/ https://unstract.com/blog/pdf-hell-and-practical-rag-applica...

gibsonf1 1y ago

We[1] Create "Units of Thought" from PDF's and then work with those for further discovery where a "Unit of Thought" is any paragraph, title, note heading - something that stands on its own semantically. We then create a hierarchy of objects from that pdf in the database for search and conceptual search - all at scale. [1] https://graphmetrix.com/trinpod-server https://graphmetrix.com/trinpod-server https://trinapp.com https://trinapp.com

IIAOPSW 1y ago
I'm tempted to try it. My use case right now is a set of documents which are annual financial and statutory disclosures of a large institution. Every year they are formatted / organized slightly differently which makes it enormously tedious to manually find and compare the same basic section from one year to another, but they are consistent enough to recognize analogous sections from different years due to often reusing verbatim quotes or highly specific key words each time. What I really want to do is take all these docs and just reorder all the content such that I can look at page n (or section whatever) scrolling down and compare it between different years by scrolling horizontally. Ideally with changes from one year to the next highlighted. Can your product do this?
- gibsonf1 1y ago
  Probably without too much difficulty. If you have a sample to confirm, that would be great. frederick @ graphmetrix . com

kbyatnal 1y ago

"PDF to Text" is a bit simplified IMO. There's actually a few class of problems within this category: 1. reliable OCR from documents (to index for search, feed into a vector DB, etc) 2. structured data extraction (pull out targeted values) 3. end-to-end document pipelines (e.g. automate mortgage applications) Marginalia needs to solve problem #1 (OCR), which is luckily getting commoditized by the day thanks to models like Gemini Flash. I've now seen multiple companies replace their OCR pipelines with Flash for a fraction of the cost of previous solutions, it's really quite remarkable. Problems #2 and #3 are much more tricky. There's still a large gap for businesses in going from raw OCR outputs —> document pipelines deployed in prod for mission-critical use cases. LLMs and VLMs aren't magic, and anyone who goes in expecting 100% automation is in for a surprise. You still need to build and label datasets, orchestrate pipelines (classify -> split -> extract), detect uncertainty and correct with human-in-the-loop, fine-tune, and a lot more. You can certainly get close to full automation over time, but it's going to take time and effort. The future is definitely moving in this direction though. Disclaimer: I started a LLM doc processing company to help companies solve problems in this space (https://extend.ai https://extend.ai)

varunneal 1y ago
I've been hacking away at trying to process PDFs into Markdown, having encountered similar obstacles to OP regarding header detection (and many other issues). OCR is fantastic these days but maintaining a global structure to the document is much trickier. Consistent HTML seems still out of reach for large documents. I'm having half-decent results with Markdown using multiple passes of an LLM to extract document structure and feeding it in contextually for page-by-pass extraction.
- dstryr 1y ago
  Give this project a try. I've been using it with promising results. https://github.com/matthsena/AlcheMark https://github.com/matthsena/AlcheMark
  - aorth 1y ago
    I tried with one PDF and was surprised to see it connect to some cloud service: 2025-05-14 07:58:49,373 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): openaipublic.blob.core.windows.net:443 2025-05-14 07:58:50,446 - urllib3.connectionpool - DEBUG - https://openaipublic.blob.core.windows.net:443 "GET /encodings/o200k_base.tiktoken HTTP/1.1" 200 361 3922 The project's README doesn't mention that anywhere...

anonu 1y ago

They should called it NDF - Non-Portable Document Format.

0xml 1y ago
Maybe Printable Document Format.

dobraczekolada 1y ago

Reminds me of github.com/docwire/docwire

90s_dev 1y ago

Have any of you ever thought to yourself, this is new and interesting, and then vaguely remembered that you spent months or years becoming an expert at it earlier in life but entirely forgot it? And in fact large chunks of the very interesting things you've done just completely flew out of your mind long ago, to the point where you feel absolutely new at life, like you've accomplished relatively nothing, until something like this jars you out of that forgetfulness? I definitely vaguely remember doing some incredibly cool things with PDFs and OCR about 6 or 7 years ago. Some project comes to mind... google tells me it was "tesseract" and that sounds familiar.

deleted 1y ago
[deleted]
downboots 1y ago
No different than a fire ant whose leaf got knocked over by the wind and it moved on to the next.
- 90s_dev 1y ago
  Well I sure do feel different than a fire ant.
  - downboots 1y ago
    anttention is all we have
    - 90s_dev 1y ago
      Not true, I also have a nice cigar waiting for the rain to go away.
      - 90s_dev 1y ago
        Hmm, it's gone now. Well I used to have one anyway.
1y ago

PeterStuer 1y ago

I guess I'm lucky the PDF's I need to process are mostly rather dull unadventurous layouts. So far I've had great success using docling.

keybored 1y ago

For people who want people to read their documents[1] they should have their PDF point to a more digital-friendly format, an alt document. Looks like you’ve found my PDF. You might want this version instead: PDFs are often subpar. Just see the first example: standard Latex serif section title. I mean, PDFs often aren’t even well-typeset for what they are (dead-tree simulations). [1] No sarcasm or truism. Some may just want to submit a paper to whatever publisher and go through their whole laundry list of what a paper ought to be. Wide dissemanation is not the point.

1vuio0pswjnm7 1y ago

Below is a PDF. It is a .txt file. I can save it with a .pdf extension and open it in a PDF viewer. I can make changes in a text editor. For example, by editing this text file, I can change the text displayed on the screen when the PDF is opened, the font, font size, line spacing, the maximum characters per line, number of lines per page, the paper width and height, as well as portrait versus landscape mode. %PDF-1.4 1 0 obj << /CreationDate (D:2025) /Producer >> endobj 2 0 obj << /Type /Catalog /Pages 3 0 R >> endobj 4 0 obj << /Type /Font /Subtype /Type1 /Name /F1 /BaseFont /Times-Roman >> endobj 5 0 obj << /Font << /F1 4 0 R >> /ProcSet [ /PDF /Text ] >> endobj 6 0 obj << /Type /Page /Parent 3 0 R /Resources 5 0 R /Contents 7 0 R >> endobj 7 0 obj << /Length 8 0 R >> stream BT /F1 50 Tf 1 0 0 1 50 752 Tm 54 TL (PDF is)' ((a) a text format)' ((b) a graphics format)' ((c) (a) and (b).)' ()' ET endstream endobj 8 0 obj 53 endobj 3 0 obj << /Type /Pages /Count 1 /MediaBox [ 0 0 612 792 ] /Kids [ 6 0 R ] >> endobj xref 0 9 0000000000 65535 f 0000000009 00000 n 0000000113 00000 n 0000000514 00000 n 0000000162 00000 n 0000000240 00000 n 0000000311 00000 n 0000000391 00000 n 0000000496 00000 n trailer << /Size 9 /Root 2 0 R /Info 1 0 R >> startxref 599 %%EOF

swsieber 1y ago
It can also have embedded binary streams. It was not made for text. It was made for layout and graphics. You give nice examples, but each of those lines could have been broken up into one call per character, or per word, even out of order.
- hnick 1y ago
  It can also use fonts which map glyphs via characters which do not represent the final visual item e.g. "PDF" could be "1#F" and you only really know what it looks like by rendering then viewing/OCR. A nice file won't, but sometimes the best work is in not dealing with nice things.
  - 90s_dev 1y ago

ljlolel 1y ago

Mistral OCR has best in class doing document understanding https://mistral.ai/news/mistral-ocr https://mistral.ai/news/mistral-ocr

nicodjimenez 1y ago

Check out mathpix.com. We handle complex tables, complex math, diagrams, rotated tables, and much more, extremely accurately. Disclaimer: I'm the founder.

bickfordb 1y ago

Maybe it's time for new document formats and browsers that neatly separate content, presentation and UI layers? PDF and HTML are 20+ years old and it's often difficult to extract information from either let alone author a browser.

rrr_oh_man 1y ago
Yes, but I'm sure they're out there somewhere (https://xkcd.com/927/ https://xkcd.com/927/)
- TRiG_Ireland 1y ago
  Open XML Paper Specification is an XML-based format intended to compete with PDF. Unlike PDF, it is purely static: no scripting. Also unlike PDF, I've never seen it actually used in the wild.

smcleod 1y ago

Definitely recommend docling for this. https://docling-project.github.io/docling/ https://docling-project.github.io/docling/

elpalek 1y ago

Recently tested a (non-english) pdf ocr with Gemini 2.5 Pro. First, directly ask it to extract text from pdf. Result: random text blob, not useable. Second, I converted pdf into pages of jpg. Gemini performed exceptional. Near perfect text extraction with intact format in markdown. Maybe there's internal difference when processing pdf vs jpg inside the model.

jagged-chisel 1y ago
Model isn’t rendering the PDF probably, just looking in the file for text.

noosphr 1y ago

I've worked on this in my day job: extracting _all_ relevant information from a financial services PDF for a bert based search engine. The only way to solve that is with a segmentation model followed by a regular OCR model and whatever other specialized models you need to extract other types of data. VLM aren't ready for prime time and won't be for a decade on more. What worked was using doclaynet trained YOLO models to get the areas of the document that were text, images, tables or formulas: https://github.com/DS4SD/DocLayNet https://github.com/DS4SD/DocLayNet if you don't care about anything but text you can feed the results into tesseract directly (but for the love of god read the manual). Congratulations, you're done. Here's some pre-trained models that work OK out of the box: https://github.com/ppaanngggg/yolo-doclaynet https://github.com/ppaanngggg/yolo-doclaynet I found that we needed to increase the resolution from ~700px to ~2100px horizontal for financial data segmentation. VLMs on the other hand still choke on long text and hallucinate unpredictably. Worse they can't understand nested data. If you give _any_ current model nothing harder than three nested rectangles with text under each they will not extract the text correctly. Given that nested rectangles describes every table no VLM can currently extract data from anything but the most straightforward of tables. But it will happily lie to you that it did - after all a mining company should own a dozen bulldozers right? And if they each cost $35.000 it must be an amazing deal they got, right?

yfontana 1y ago
I've been working on extracting text from some 20 million PDFs, with just about every type of layout you can imagine. We're using a similar approach (segmentation / OCR), but with PyMuPDF. The full extract is projected to run for several days on a GPU cluster, at a cost of like 20-30k (can't remember the exact number but it's in that ballpark). When you can afford this kind of compute, text extraction from PDFs isn't quite a fully solved problem, but we're most of the way there. What the article in the OP tries to do is, as far as I understand, somewhat different. It's trying to use much simpler heuristics to get acceptable results cheaper and faster, and this is definitely an open issue.
cess11 1y ago
That looks like a pretty good starting point, thanks. I've been dabbling in vision models but need a much higher degree of accuracy than they seem able to provide, opting instead for more traditional techniques and handling errors manually.

Sharlin 1y ago

Some of the unsung heroes of the modern age are the programmers who, through what must have involved a lot of weeping and gnashing of teeth, have managed to implement the find, select, and copy operations in PDF readers.

patrick41638265 1y ago

Good old https://linux.die.net/man/1/pdftotext https://linux.die.net/man/1/pdftotext and a little Python on top of its output will get you a long way if your documents are not too crazy. I use it to parse all my bank statements into an sqlite database for analysis.

coolcase 1y ago

Tried extracting data from a newspaper. It is really hard. What is a headline and which headline belongs to which paragraphs? Harder than you think! And chucking it as is into OpenAI was no good at all. Manually dealing with coordinates from OCR was better but not perfect.

rekoros 1y ago

I've been using Azure's "Document Intelligence" thingy (prebuilt "read" model) to extract text from PDFs with pretty good results [1]. Their terminology is so bad, it's easy to dismiss the whole thing for another Microsoft pile, but it actually, like, for real, works. [1] https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/prebuilt/read?view=doc-intel-4.0.0&tabs=sample-code https://learn.microsoft.com/en-us/azure/ai-services/document...

incanus77 1y ago

I did some contract work some years back with a company who had a desktop product (for Mac) that would apply some smarts to strip out extraneous things on pages while printing (such as ads on webpages) as well as try to avoid the case where only a line or two was printed on a page, wasting paper. It initially was getting into things at the PostScript layer, which unsurprisingly was horrifying, but eventually worked on PDFs. This required finding and interpreting various textual parts of the passed documents and was a pretty big technical challenge. While I'm not convinced it was viable at the business level, it feels like something platform/OS companies could focus on to have a measurable environmental and cost overhead impact.

viking2917 1y ago

coincidentally, posted this over on Show HN today. OCR workbench, AI OCR & editing tools for OCRing old / hard documents. https://news.ycombinator.com/item?id=43976450 https://news.ycombinator.com/item?id=43976450. Tesseract works fine for modern text documents, but it fails badly on older docs (e.g. colonial american, etc)

gerdesj 1y ago

PDF is a display format. It is optimised for eyeballs and printers. There has been some feature creep. It is a rubbish machine data transfer mechanism but really good for humans and say storing a page of A4 (letter for the US). So, you start off with the premise that a .pdf stores text and you want that text. Well that's nice: grow some eyes! Otherwise, you are going to have to get to grips with some really complicated stuff. For starters, is the text ... text or is it an image? Your eyes don't care and will just work (especially when you pop your specs back on) but your parser is probably seg faulting madly. It just gets worse. PDF is for humans to read. Emulate a human to read a PDF.

amai 1y ago
That's it. I could not have said it better.

TZubiri 1y ago

As someone who has worked on this FT. (S&P, parsing of financial disclosures) The solution is OCR. Don't fuck with internal file format. PDF is designed to print/display stuff, not to be parseable by machines.

fracus 1y ago

Why hasn't the PDF standard been replaced or revised to require the text in meta form? Seems like a no brainer.

msephton 1y ago
It's already supported, but it's an optional feature, so it's up to the app/developer/author.

remram 1y ago

I built a simple OSS tool for qualitative data analysis, which needs to turn uploaded documents into text (stripped HTML). PDFs have been a huge problem from day one. I have investigated many tools, but two-column layouts and footers etc often still mess up the content. It's hard to convince my (often non-technical) users that this is a difficult problem.

bartread 1y ago
Try Poppler’s pdftohtml command line tool. For me that seems to do a good job of spitting out multi-column text in the right order. Then you have the much easier task of extracting the text from the HTML. Also, if it does come out in the wrong order for any pages you can analyse element coordinates to figure out which column each chunk of text belongs in. (Note that you may have to deal with sub-columns if tables are present in any columns. I’ve never had this in my data but you may also find blocks that span across more than one column, either in whole or in part.) They also have a pdftotext tool that may do the job for you if you disable its layout option. If you run it with the layout option enabled you’ll find it generates multi-column text in the output, as it tries to closely match the layout of the input PDF. I think the pdftohtml tool is probably the way to go just because the extra metadata on each element is probably going to be helpful in determining how to treat that element, and it’s obviously relatively straightforward to strip out the HTML tags to extract plain text.

trevor-e 1y ago

Having built some toy parsers for PDF files in the past it was a huge wtf moment for me when I realized how the format works. With that in mind, it's even more puzzling how it's used often in text-heavy cases. I always think about the invoicing use-case: digital systems should be able to easy extract data from the file while still being formatted visually for humans. It seems like the tech world would be much better off if we migrated to a better format.

Uvix 1y ago
XML+XSLT was almost this, but unfortunately browsers no longer support it for local XML files, only ones on a remote server.

Klaus_ 1y ago

[dead]

herodotus 1y ago

This is mostly what I worked on for many years at Apple with reasonable success. The main secret was to accept that everything was geometry, and use cluster analysis to try to distinguish between word gaps and letter gaps. On many PDF documents, it works really well, but there are so many different kinds of PDF documents that there are always cases were the results are not that great. If I were to do it today, I would stick with geometry, avoid OCR completely, but use machine learning. One big advantage for machine learning is that I could use existing tools to generate PDFs from known text, so that the training phase could be completly automatic. (Here is Bertrand Serlet announcing the feature at WWDC in 2009: https://youtu.be/FTfChHwGFf0?si=wNCfI9wZj1aj9rY7&t=308 https://youtu.be/FTfChHwGFf0?si=wNCfI9wZj1aj9rY7&t=308)

lewtun 1y ago

> The absolute best way of doing this is these days is likely through a vision based machine learning model, but that is an approach that is very far away from scaling to processing hundreds of gigabytes of PDF files off a single server with no GPU. SmolDocling is pretty fast and the ONNX weights can be scaled to many CPUs: https://huggingface.co/ds4sd/SmolDocling-256M-preview https://huggingface.co/ds4sd/SmolDocling-256M-preview Not sure what time scale the author had in mind for processing GBs of PDFs, but the future might be closer than “very far away”

snehanairdoc 1y ago

This was a great read. You've done an excellent job breaking down what makes PDFs so uniquely annoying to work with. People often underestimate how much of the “document-ness” (like headings, paragraphs, tables) is just visual, with no underlying semantic structure. We ran into many of the same challenges while working on Docsumo, where we process business documents like invoices, bank statements, and scanned PDFs. In real-world use cases, things get even messier: inconsistent templates, rotated scans, overlapping text, or documents generated by ancient software with no tagging at all. One thing we’ve found helpful (in addition to heuristics like font size/weight and spacing) is combining layout parsing with ML models trained to infer semantic roles (like "header", "table cell", "footer", etc.). It’s far from perfect, but it helps bridge the gap between how the document looks and what it means. Really appreciate posts like this. PDF wrangling is a dark art more people should talk about.

hilbert42 1y ago

"It doesn’t have text in the way you might think of it, but more of a mapping of glyphs to coordinates on “paper”." I've often had trouble extracting text from PDFs, it's time consuming and messy, so a quick question. The PDF format works pretty well for what it does but it's now pretty ancient, so does anyone know if there's any newer format on the horizon that could be a next-generation replacement that would make it much easier to extract its data and export it to another format (say, docx, odt, etc.)?

drguthals 1y ago

PDF -> Useful Information is what Tensorlake does (https://tensorlake.ai https://tensorlake.ai) Because PDFs are so dominate and yet each one has information in more than just text (tables, images, formulas, hand-writing, strike-throughs even), we (as devs) need to be able have tools that understand the contents, not just "read" them. Full disclosure...I work there

nemosaltat 1y ago

`uv run marker_single “path/to/file.pdf”` auto dumps markdown, images, and a json map of the PDF to a folder in the venv and gives you the path. It’s like magic.