7 ms·

Yeah, getting text - even structured text - out of PDFs is no picnic. Scraping a table out of an HTML document is often straightforward even on sites that use t

Yeah, getting text - even structured text - out of PDFs is no picnic. Scraping a table out of an HTML document is often straightforward even on sites that use the "everything's a <div>" (anti-)pattern, and especially on sites that use more semantically useful elements, like <table>. Not so PDFs. I'm far from an expert on the format, so maybe there is some semantic support in there, but I've seen plenty of PDFs where tables are simply an loose assemblage of graphical and text elements that, only when rendered, are easily discernible as a table because they're positioned in such a way that they render as a table. I've actually had decent luck extracting tabular data from PDFS by converting the PDFs to HTML using the Poppler PDF utils, then finding the expected table header, and then using the x-coordinate of the HTML elements for each value within the table to work out columns, and extract values for each rows. It's kind of groaty but it seems reliable for what I need. Certainly much moreso than going via formatted plaintext, which has issues with inconsistent spacing, and the insertion of newlines into the middle of rows.

j45 1y ago
PDFs inherently are a markup / xml format, the standard is available to learn from. It's possible to create the same PDF in many, many, many ways. Some might lean towards exporting a layout containing text and graphics from a graphics suite. Others might lean towards exporting text and graphics from a word processor, which is words first. The lens of how the creating app deals with information is often something that has input on how the PDF is output. If you're looking for an off the shelf utility that is surprisingly decent at pulling structured data from PDFs, tools like cisdem have already solved enough of it for local users. Lots of tools like this out there, many do promise structured data support but it needs to match what you're up to.
- layer8 1y ago
  > PDFs inherently are a markup / xml format This is false. PDFs are an object graph containing imperative-style drawing instructions (among many other things). There’s a way to add structural information on top (akin to an HTML document structure), but that’s completely optional and only serves as auxiliary metadata, it’s not at the core of the PDF format.
  - davidthewatson 1y ago
    Thanks for your comment. Indeed. Therein lies the rub. Why? Because no matter the fact that I've spent several years of my latent career crawling and parsing and outputting PDF data, I see now that pointing my LLLM stack at a directory of *.pdf just makes the invisible encoding of the object graph visible. It's a skeptical science. The key transclusion may be to move from imperative to declarative tools or conditional to probabilistic tools, as many areas have in the last couple decades. I've been following John Sterling's ocaml work for a while on related topics and the ideas floating around have been a good influence on me in forests and their forester which I found resonant given my own experience: https://www.jonmsterling.com/index/index.xml https://www.jonmsterling.com/index/index.xml https://github.com/jonsterling/forest https://github.com/jonsterling/forest I was gonna email john and ask whether it's still being worked on as I hope so, but I brought it up this morning as a way out of the noise that imperative programming PDF has been for a decade or more where turtles all the way down to the low-level root cause libraries mean that the high level imperative languages often display the exact same bugs despite significant differences as to what's being intended in the small on top of the stack vs the large on the bottom of the stack. It would help if "fitness for a particular purpose" decisions were thoughtful as to publishing and distribution but as the CFO likes to say, "Dave, that ship has already sailed." Sigh. ¯\_(ツ)_/¯
  - j45 1y ago
    I appreciate the clarification. Should have been more precise with my terminology. That being said, I think I'm talking about the forest of PDFs. When I said PDFs have a "markup-like structure," I was talking from my experience manually writing PDFs from scratch using Adobe's spec. PDFs definitely have a structured, hierarchical format with nested elements that looks a lot like markup languages conceptually. The objects have a structure comparable to DOM-like structures - there's clear parent-child relationships just like in markup languages. Working with tags like "<<" and ">>" feels similar to markup tags when hand coding them. This is an article that highlights what I have seen (much cleaner PDF code): "The Structure of a PDF File" (https://medium.com/@jberkenbilt/the-structure-of-a-pdf-file-6f08114a58f6 https://medium.com/@jberkenbilt/the-structure-of-a-pdf-file-...) which says: "There are several types of objects. If you are familiar with JSON, YAML, or the object model in any reasonably modern programming language, this will seem very familiar to you... A PDF object may have one of the following types: String, Number, Boolean, Null, Name, Array, Dictionary..." This structure with dictionaries in "<<" and ">>" and arrays in brackets really gave me markup vibes when coding to the spec (https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandards/PDF32000_2008.pdf https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandard...). While PDFs are an object graph with drawing instructions like you said, the structure itself looks a lot like markup formats. Might be just a difference in choosing to focus on the forest vs the trees. That hierarchical structure is why different PDF creation methods can make such varied document structures, which is exactly why text extraction is so tricky. Learning to hand code PDFs in many ways, lets you learn to read and unravel them a little differently, maybe even a bit easier.
- jimjimjim 1y ago
  uh. There is very little XML and the spec is a thousand pages long.
  - j45 1y ago
    Clarified above - referring to the visual side of coding PDFs by hand. https://medium.com/@jberkenbilt/the-structure-of-a-pdf-file-6f08114a58f6 https://medium.com/@jberkenbilt/the-structure-of-a-pdf-file-...
yxhuvud 1y ago
My favorite is (official, governmental) documents that has one set of text that is rendered, and a totally different set of text that you get if you extract the text the normal way..
hermitcrab 1y ago
I am hoping at some point to be able to extract tabular data from PDFs for my data wrangling software. If anyone knows of a library that can extract tables from PDFs, can be inegrated into a C++ app and is free or less than a few hundred $, please let me know!
- ______ 1y ago
  pdfplumber is great for table extraction but it is python
  - hermitcrab 1y ago
    Thanks, but I prefer to keep everything C++ for simplicity and speed.
spacecaps 1y ago
I was irritated that I couldn't extract data from PDFs in a similar way to web pages + BeautifulSoup, so I built a library that (kind of) does just that[0]. It does a bunch of other nonsense, but the main goal is a more "human" way of interacting, e.g. `page.find('text:bold:contains("Summary").below().extract_text()`. And since every PDF is its own bespoke nightmare, I'm also trying to build up a collection of awful-to-extract-data-from examples to serve as the foundation for a how-to library[1]. [0] https://jsoma.github.io/natural-pdf/ https://jsoma.github.io/natural-pdf/ [1] https://badpdfs.com/ https://badpdfs.com/

j45 1y ago

PDFs inherently are a markup / xml format, the standard is available to learn from. It's possible to create the same PDF in many, many, many ways. Some might lean towards exporting a layout containing text and graphics from a graphics suite. Others might lean towards exporting text and graphics from a word processor, which is words first. The lens of how the creating app deals with information is often something that has input on how the PDF is output. If you're looking for an off the shelf utility that is surprisingly decent at pulling structured data from PDFs, tools like cisdem have already solved enough of it for local users. Lots of tools like this out there, many do promise structured data support but it needs to match what you're up to.

layer8 1y ago
> PDFs inherently are a markup / xml format This is false. PDFs are an object graph containing imperative-style drawing instructions (among many other things). There’s a way to add structural information on top (akin to an HTML document structure), but that’s completely optional and only serves as auxiliary metadata, it’s not at the core of the PDF format.
- davidthewatson 1y ago
  Thanks for your comment. Indeed. Therein lies the rub. Why? Because no matter the fact that I've spent several years of my latent career crawling and parsing and outputting PDF data, I see now that pointing my LLLM stack at a directory of *.pdf just makes the invisible encoding of the object graph visible. It's a skeptical science. The key transclusion may be to move from imperative to declarative tools or conditional to probabilistic tools, as many areas have in the last couple decades. I've been following John Sterling's ocaml work for a while on related topics and the ideas floating around have been a good influence on me in forests and their forester which I found resonant given my own experience: https://www.jonmsterling.com/index/index.xml https://www.jonmsterling.com/index/index.xml https://github.com/jonsterling/forest https://github.com/jonsterling/forest I was gonna email john and ask whether it's still being worked on as I hope so, but I brought it up this morning as a way out of the noise that imperative programming PDF has been for a decade or more where turtles all the way down to the low-level root cause libraries mean that the high level imperative languages often display the exact same bugs despite significant differences as to what's being intended in the small on top of the stack vs the large on the bottom of the stack. It would help if "fitness for a particular purpose" decisions were thoughtful as to publishing and distribution but as the CFO likes to say, "Dave, that ship has already sailed." Sigh. ¯\_(ツ)_/¯
- j45 1y ago
  I appreciate the clarification. Should have been more precise with my terminology. That being said, I think I'm talking about the forest of PDFs. When I said PDFs have a "markup-like structure," I was talking from my experience manually writing PDFs from scratch using Adobe's spec. PDFs definitely have a structured, hierarchical format with nested elements that looks a lot like markup languages conceptually. The objects have a structure comparable to DOM-like structures - there's clear parent-child relationships just like in markup languages. Working with tags like "<<" and ">>" feels similar to markup tags when hand coding them. This is an article that highlights what I have seen (much cleaner PDF code): "The Structure of a PDF File" (https://medium.com/@jberkenbilt/the-structure-of-a-pdf-file-6f08114a58f6 https://medium.com/@jberkenbilt/the-structure-of-a-pdf-file-...) which says: "There are several types of objects. If you are familiar with JSON, YAML, or the object model in any reasonably modern programming language, this will seem very familiar to you... A PDF object may have one of the following types: String, Number, Boolean, Null, Name, Array, Dictionary..." This structure with dictionaries in "<<" and ">>" and arrays in brackets really gave me markup vibes when coding to the spec (https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandards/PDF32000_2008.pdf https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandard...). While PDFs are an object graph with drawing instructions like you said, the structure itself looks a lot like markup formats. Might be just a difference in choosing to focus on the forest vs the trees. That hierarchical structure is why different PDF creation methods can make such varied document structures, which is exactly why text extraction is so tricky. Learning to hand code PDFs in many ways, lets you learn to read and unravel them a little differently, maybe even a bit easier.
jimjimjim 1y ago
uh. There is very little XML and the spec is a thousand pages long.
- j45 1y ago
  Clarified above - referring to the visual side of coding PDFs by hand. https://medium.com/@jberkenbilt/the-structure-of-a-pdf-file-6f08114a58f6 https://medium.com/@jberkenbilt/the-structure-of-a-pdf-file-...

yxhuvud 1y ago

My favorite is (official, governmental) documents that has one set of text that is rendered, and a totally different set of text that you get if you extract the text the normal way..

hermitcrab 1y ago

I am hoping at some point to be able to extract tabular data from PDFs for my data wrangling software. If anyone knows of a library that can extract tables from PDFs, can be inegrated into a C++ app and is free or less than a few hundred $, please let me know!

______ 1y ago
pdfplumber is great for table extraction but it is python
- hermitcrab 1y ago
  Thanks, but I prefer to keep everything C++ for simplicity and speed.

spacecaps 1y ago

I was irritated that I couldn't extract data from PDFs in a similar way to web pages + BeautifulSoup, so I built a library that (kind of) does just that[0]. It does a bunch of other nonsense, but the main goal is a more "human" way of interacting, e.g. `page.find('text:bold:contains("Summary").below().extract_text()`. And since every PDF is its own bespoke nightmare, I'm also trying to build up a collection of awful-to-extract-data-from examples to serve as the foundation for a how-to library[1]. [0] https://jsoma.github.io/natural-pdf/ https://jsoma.github.io/natural-pdf/ [1] https://badpdfs.com/ https://badpdfs.com/