3 ms·

PDF is a pretty interesting format. The spec is actually a great read. It's amazing how many features they've needed to add over the years to support everyone's

by aidos 9y ago

PDF is a pretty interesting format. The spec is actually a great read. It's amazing how many features they've needed to add over the years to support everyone's use cases. It's a display format that doesn't have a whole lot of semantic meaning for the most part. Often every character is individually placed so even extracting words is a pain. It's insane that OCR (which it sounds like this uses) is the easiest way to deal with extraction. I highly recommend having a look inside a couple of pdfs to see how they look. I've posted about this before but the trick is to expand the streams. mutool clean -d in.pdf out.pdf

petters 9y ago
Each character needs to be individually placed for the best possible typesetting (e.g with microtype). Nevertheless, copying text from a PDF works in many cases, so there has to be some support for that, no? Or is everything done in the reader?
- aidos 9y ago
  Text is a funny thing in PDF. Sometimes, there's real text broken into lines and then you're pretty good. Often it's a subset font where the codes used in the script don't correspond to their visual glyphs. When that happens there might be a unicode map that lets you know which internal char maps to which external char (used for copying text from the PDF by viewers). Sometimes that's missing and you can rebuild it from other encoding information attached to the font. Other times you can't get the relationship and all you're left with is the randomly selected codes for each character — at that point it's not too dissimilar from a simple Enigma machine problem I guess. I sometimes see documents in which the same font has been subset again and again, once for each word. If you have a unicode map for each one, that's fine, if not it's not going to be much fun. In that case every word is going to look like random characters in the PDF and those character-glyph relationships are going to change from word to word. Other times the glyphs are rendered as vector paths at write time and you're down to trying to find the character from the outline it the shape. I deal with this a lot and there are common patterns but normally each glyph will be broken into several bits itself, so you have to find which bits go with which glyphs before you even start. Does anyone have a technical paper to hand, we could crack it open and take a look for fun. EDIT If you're trying to reliably convert everything, in a way you're better off catering for the lowest common denominator so you can do it consistently. Here, that means assuming you don't have any raw text data to work with and you have to do everything with image recognition. Either way, it's a fun problem.

by aidos 9y ago

petters 9y ago

Each character needs to be individually placed for the best possible typesetting (e.g with microtype). Nevertheless, copying text from a PDF works in many cases, so there has to be some support for that, no? Or is everything done in the reader?

aidos 9y ago
Text is a funny thing in PDF. Sometimes, there's real text broken into lines and then you're pretty good. Often it's a subset font where the codes used in the script don't correspond to their visual glyphs. When that happens there might be a unicode map that lets you know which internal char maps to which external char (used for copying text from the PDF by viewers). Sometimes that's missing and you can rebuild it from other encoding information attached to the font. Other times you can't get the relationship and all you're left with is the randomly selected codes for each character — at that point it's not too dissimilar from a simple Enigma machine problem I guess. I sometimes see documents in which the same font has been subset again and again, once for each word. If you have a unicode map for each one, that's fine, if not it's not going to be much fun. In that case every word is going to look like random characters in the PDF and those character-glyph relationships are going to change from word to word. Other times the glyphs are rendered as vector paths at write time and you're down to trying to find the character from the outline it the shape. I deal with this a lot and there are common patterns but normally each glyph will be broken into several bits itself, so you have to find which bits go with which glyphs before you even start. Does anyone have a technical paper to hand, we could crack it open and take a look for fun. EDIT If you're trying to reliably convert everything, in a way you're better off catering for the lowest common denominator so you can do it consistently. Here, that means assuming you don't have any raw text data to work with and you have to do everything with image recognition. Either way, it's a fun problem.