4 ms·

PDFs inherently are a markup / xml format, the standard is available to learn from. It's possible to create the same PDF in many, many, many ways. Some might

by j45 1y ago

PDFs inherently are a markup / xml format, the standard is available to learn from. It's possible to create the same PDF in many, many, many ways. Some might lean towards exporting a layout containing text and graphics from a graphics suite. Others might lean towards exporting text and graphics from a word processor, which is words first. The lens of how the creating app deals with information is often something that has input on how the PDF is output. If you're looking for an off the shelf utility that is surprisingly decent at pulling structured data from PDFs, tools like cisdem have already solved enough of it for local users. Lots of tools like this out there, many do promise structured data support but it needs to match what you're up to.

layer8 1y ago
> PDFs inherently are a markup / xml format This is false. PDFs are an object graph containing imperative-style drawing instructions (among many other things). There’s a way to add structural information on top (akin to an HTML document structure), but that’s completely optional and only serves as auxiliary metadata, it’s not at the core of the PDF format.
- davidthewatson 1y ago
  Thanks for your comment. Indeed. Therein lies the rub. Why? Because no matter the fact that I've spent several years of my latent career crawling and parsing and outputting PDF data, I see now that pointing my LLLM stack at a directory of *.pdf just makes the invisible encoding of the object graph visible. It's a skeptical science. The key transclusion may be to move from imperative to declarative tools or conditional to probabilistic tools, as many areas have in the last couple decades. I've been following John Sterling's ocaml work for a while on related topics and the ideas floating around have been a good influence on me in forests and their forester which I found resonant given my own experience: https://www.jonmsterling.com/index/index.xml https://www.jonmsterling.com/index/index.xml https://github.com/jonsterling/forest https://github.com/jonsterling/forest I was gonna email john and ask whether it's still being worked on as I hope so, but I brought it up this morning as a way out of the noise that imperative programming PDF has been for a decade or more where turtles all the way down to the low-level root cause libraries mean that the high level imperative languages often display the exact same bugs despite significant differences as to what's being intended in the small on top of the stack vs the large on the bottom of the stack. It would help if "fitness for a particular purpose" decisions were thoughtful as to publishing and distribution but as the CFO likes to say, "Dave, that ship has already sailed." Sigh. ¯\_(ツ)_/¯
- j45 1y ago
  I appreciate the clarification. Should have been more precise with my terminology. That being said, I think I'm talking about the forest of PDFs. When I said PDFs have a "markup-like structure," I was talking from my experience manually writing PDFs from scratch using Adobe's spec. PDFs definitely have a structured, hierarchical format with nested elements that looks a lot like markup languages conceptually. The objects have a structure comparable to DOM-like structures - there's clear parent-child relationships just like in markup languages. Working with tags like "<<" and ">>" feels similar to markup tags when hand coding them. This is an article that highlights what I have seen (much cleaner PDF code): "The Structure of a PDF File" (https://medium.com/@jberkenbilt/the-structure-of-a-pdf-file-6f08114a58f6 https://medium.com/@jberkenbilt/the-structure-of-a-pdf-file-...) which says: "There are several types of objects. If you are familiar with JSON, YAML, or the object model in any reasonably modern programming language, this will seem very familiar to you... A PDF object may have one of the following types: String, Number, Boolean, Null, Name, Array, Dictionary..." This structure with dictionaries in "<<" and ">>" and arrays in brackets really gave me markup vibes when coding to the spec (https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandards/PDF32000_2008.pdf https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandard...). While PDFs are an object graph with drawing instructions like you said, the structure itself looks a lot like markup formats. Might be just a difference in choosing to focus on the forest vs the trees. That hierarchical structure is why different PDF creation methods can make such varied document structures, which is exactly why text extraction is so tricky. Learning to hand code PDFs in many ways, lets you learn to read and unravel them a little differently, maybe even a bit easier.
  - layer8 1y ago
    Markup is only indirectly related to hierarchical structure. “Markup” means that there is text that is being “marked up” with additional attributes (styling, structure information, metadata, …). This is how HTML and XML work, and also languages like TeX, Troff, and Markdown. For example, in the text “this is some text”, you can mark up the word “some” as being emphasized, as in “this is <em>some</em> text”. The general principle is that the base content is plain text, which is augmented with markup information, which may or may not have hierarchical aspects. You can simply strip away the markup again and recover just the text. That’s not at all how PDF works, however. You cite a comparison to JSON and YAML. Those are not markup languages (despite what YAML originally was an abbreviation for, see [0]). (HTML also isn’t DOM.) [0] https://stackoverflow.com/a/18928199 https://stackoverflow.com/a/18928199
- bartread 1y ago
  > but that’s completely optional and only serves as auxiliary metadata, it’s not at the core of the PDF format. This is what I kind of suspected but, as I said in my original comment, I'm not an expert and for the PDFs I'm reading I didn't need to delve further because that metadata simply isn't in there (although, boy do I wish it was) so I needed to use a different approach. As soon as I realised what I had was purely presentation I knew it was going to be a bit grim.
jimjimjim 1y ago
uh. There is very little XML and the spec is a thousand pages long.
- j45 1y ago
  Clarified above - referring to the visual side of coding PDFs by hand. https://medium.com/@jberkenbilt/the-structure-of-a-pdf-file-6f08114a58f6 https://medium.com/@jberkenbilt/the-structure-of-a-pdf-file-...

by j45 1y ago

layer8 1y ago

> PDFs inherently are a markup / xml format This is false. PDFs are an object graph containing imperative-style drawing instructions (among many other things). There’s a way to add structural information on top (akin to an HTML document structure), but that’s completely optional and only serves as auxiliary metadata, it’s not at the core of the PDF format.

davidthewatson 1y ago
Thanks for your comment. Indeed. Therein lies the rub. Why? Because no matter the fact that I've spent several years of my latent career crawling and parsing and outputting PDF data, I see now that pointing my LLLM stack at a directory of *.pdf just makes the invisible encoding of the object graph visible. It's a skeptical science. The key transclusion may be to move from imperative to declarative tools or conditional to probabilistic tools, as many areas have in the last couple decades. I've been following John Sterling's ocaml work for a while on related topics and the ideas floating around have been a good influence on me in forests and their forester which I found resonant given my own experience: https://www.jonmsterling.com/index/index.xml https://www.jonmsterling.com/index/index.xml https://github.com/jonsterling/forest https://github.com/jonsterling/forest I was gonna email john and ask whether it's still being worked on as I hope so, but I brought it up this morning as a way out of the noise that imperative programming PDF has been for a decade or more where turtles all the way down to the low-level root cause libraries mean that the high level imperative languages often display the exact same bugs despite significant differences as to what's being intended in the small on top of the stack vs the large on the bottom of the stack. It would help if "fitness for a particular purpose" decisions were thoughtful as to publishing and distribution but as the CFO likes to say, "Dave, that ship has already sailed." Sigh. ¯\_(ツ)_/¯
j45 1y ago
I appreciate the clarification. Should have been more precise with my terminology. That being said, I think I'm talking about the forest of PDFs. When I said PDFs have a "markup-like structure," I was talking from my experience manually writing PDFs from scratch using Adobe's spec. PDFs definitely have a structured, hierarchical format with nested elements that looks a lot like markup languages conceptually. The objects have a structure comparable to DOM-like structures - there's clear parent-child relationships just like in markup languages. Working with tags like "<<" and ">>" feels similar to markup tags when hand coding them. This is an article that highlights what I have seen (much cleaner PDF code): "The Structure of a PDF File" (https://medium.com/@jberkenbilt/the-structure-of-a-pdf-file-6f08114a58f6 https://medium.com/@jberkenbilt/the-structure-of-a-pdf-file-...) which says: "There are several types of objects. If you are familiar with JSON, YAML, or the object model in any reasonably modern programming language, this will seem very familiar to you... A PDF object may have one of the following types: String, Number, Boolean, Null, Name, Array, Dictionary..." This structure with dictionaries in "<<" and ">>" and arrays in brackets really gave me markup vibes when coding to the spec (https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandards/PDF32000_2008.pdf https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandard...). While PDFs are an object graph with drawing instructions like you said, the structure itself looks a lot like markup formats. Might be just a difference in choosing to focus on the forest vs the trees. That hierarchical structure is why different PDF creation methods can make such varied document structures, which is exactly why text extraction is so tricky. Learning to hand code PDFs in many ways, lets you learn to read and unravel them a little differently, maybe even a bit easier.
- layer8 1y ago
  Markup is only indirectly related to hierarchical structure. “Markup” means that there is text that is being “marked up” with additional attributes (styling, structure information, metadata, …). This is how HTML and XML work, and also languages like TeX, Troff, and Markdown. For example, in the text “this is some text”, you can mark up the word “some” as being emphasized, as in “this is <em>some</em> text”. The general principle is that the base content is plain text, which is augmented with markup information, which may or may not have hierarchical aspects. You can simply strip away the markup again and recover just the text. That’s not at all how PDF works, however. You cite a comparison to JSON and YAML. Those are not markup languages (despite what YAML originally was an abbreviation for, see [0]). (HTML also isn’t DOM.) [0] https://stackoverflow.com/a/18928199 https://stackoverflow.com/a/18928199
bartread 1y ago
> but that’s completely optional and only serves as auxiliary metadata, it’s not at the core of the PDF format. This is what I kind of suspected but, as I said in my original comment, I'm not an expert and for the PDFs I'm reading I didn't need to delve further because that metadata simply isn't in there (although, boy do I wish it was) so I needed to use a different approach. As soon as I realised what I had was purely presentation I knew it was going to be a bit grim.

jimjimjim 1y ago

uh. There is very little XML and the spec is a thousand pages long.

j45 1y ago
Clarified above - referring to the visual side of coding PDFs by hand. https://medium.com/@jberkenbilt/the-structure-of-a-pdf-file-6f08114a58f6 https://medium.com/@jberkenbilt/the-structure-of-a-pdf-file-...