4 ms·

Isn't it amazing how one company invented a universally spread format that takes structured data from an editor (except images obviously) and converts it into a

Isn't it amazing how one company invented a universally spread format that takes structured data from an editor (except images obviously) and converts it into a completely fucked-up unstructured form that then requires expensive voodoo magic to convert back into structured data.

sconeguy 2y ago
What would it have taken to store the plain text in some meta field in the document. Argh, so annoying.
- dkjaudyeqooe 2y ago
  PDF provide that capability, but editors don't produce it, probably because printing is though OS drivers that don't support it, or PDF generators that don't support it. Or they do support it but users don't know to check that option, or turn it off because it makes PDFs too large.
  - user_7832 2y ago
    Do you know what this field/type is called, and I’d any of the big names (MS/Adobe etc) support creating such PDFs?
    - bux93 2y ago
      OCR software like ABBY can spit out something called a "searchable PDF", which has a text layer underneath a picture of a scan. Otherwise, PDF has 'dictionaries' with arbitrary key-value pairs in them. The "Info" dictionary has some specific metadata fields like Author, and a "Font" dictionary embeds fonts, but you're free to use those dictionaries for whatever. There's also a standard to embed 'dublin core', rights management and custom metadata called XMP. Files can be embedded. You can also use comments, as PDF is a subset of postscript. When a PDF gets converted to PDF/A (by archiving software) or flattened/optimized, most of these are likely to be lost.
    - dkjaudyeqooe 2y ago
      I believe it's a "hybrid PDF" but I'm not sure if there's a further standard for merely embedding text. https://stackoverflow.com/questions/67358370/what-the-standard-used-by-a-hybrid-pdf-file https://stackoverflow.com/questions/67358370/what-the-standa...
- groby_b 2y ago
  PDF supports that just fine. It's just that many PDF publishers choose not to use that. You can lead a horse to water...
shermantanktop 2y ago
is "put this glyph at coordinate (x,y)" really what you'd call "structured"?
- dkjaudyeqooe 2y ago
  He's calling PDFs unstructured: structured editors -> unstructured PDF -> structured data
- irjustin 2y ago
  It's not the structure that allows meaningful understanding. Something that was clearly a table now becomes a bunch of glphy's physically close to eachother vs a group of other glphys but when considered as a group is a box visually separated from another group of glphys but actually part of a table.
nitwit005 2y ago
People kind of dump whatever in pdf files, so I don't think a cleaner file format would do as much as you might think. Digital fax services will generate pdf files, for example. They're just image data dumped into a pdf. Various scanners will also do so.
surfingdino 2y ago
In my experience AWS Textextract does a pretty good job without using LLMs.
shermantanktop 2y ago
PDFs began as just postscript commands stored in a file. It’s a genius hack in a way that has become a Frankenstein’s monster.
Bluestein 2y ago
... and call's it "portable", to boot.-

sconeguy 2y ago

What would it have taken to store the plain text in some meta field in the document. Argh, so annoying.

dkjaudyeqooe 2y ago
PDF provide that capability, but editors don't produce it, probably because printing is though OS drivers that don't support it, or PDF generators that don't support it. Or they do support it but users don't know to check that option, or turn it off because it makes PDFs too large.
- user_7832 2y ago
  Do you know what this field/type is called, and I’d any of the big names (MS/Adobe etc) support creating such PDFs?
  - bux93 2y ago
    OCR software like ABBY can spit out something called a "searchable PDF", which has a text layer underneath a picture of a scan. Otherwise, PDF has 'dictionaries' with arbitrary key-value pairs in them. The "Info" dictionary has some specific metadata fields like Author, and a "Font" dictionary embeds fonts, but you're free to use those dictionaries for whatever. There's also a standard to embed 'dublin core', rights management and custom metadata called XMP. Files can be embedded. You can also use comments, as PDF is a subset of postscript. When a PDF gets converted to PDF/A (by archiving software) or flattened/optimized, most of these are likely to be lost.
  - dkjaudyeqooe 2y ago
    I believe it's a "hybrid PDF" but I'm not sure if there's a further standard for merely embedding text. https://stackoverflow.com/questions/67358370/what-the-standard-used-by-a-hybrid-pdf-file https://stackoverflow.com/questions/67358370/what-the-standa...
groby_b 2y ago
PDF supports that just fine. It's just that many PDF publishers choose not to use that. You can lead a horse to water...

shermantanktop 2y ago

is "put this glyph at coordinate (x,y)" really what you'd call "structured"?

dkjaudyeqooe 2y ago
He's calling PDFs unstructured: structured editors -> unstructured PDF -> structured data
irjustin 2y ago
It's not the structure that allows meaningful understanding. Something that was clearly a table now becomes a bunch of glphy's physically close to eachother vs a group of other glphys but when considered as a group is a box visually separated from another group of glphys but actually part of a table.

nitwit005 2y ago

People kind of dump whatever in pdf files, so I don't think a cleaner file format would do as much as you might think. Digital fax services will generate pdf files, for example. They're just image data dumped into a pdf. Various scanners will also do so.

surfingdino 2y ago

In my experience AWS Textextract does a pretty good job without using LLMs.

shermantanktop 2y ago

PDFs began as just postscript commands stored in a file. It’s a genius hack in a way that has become a Frankenstein’s monster.

Bluestein 2y ago

... and call's it "portable", to boot.-