7 ms·
Cool! That's a good intro too. Many people don't realise the general weird disconnect in PDFs between real content and what you see on the screen that makes it
by aidos 8y ago
Cool! That's a good intro too.
Many people don't realise the general weird disconnect in PDFs between real content and what you see on the screen that makes it hard to recover source data. In extreme cases you have subset fonts with glyphs ordered completely differently from how they are in the original and no mapping back to the character they represent. Then the graphics stream is instructions to draw glyphs at coordinates. As you can imagine it's quite a battle to get back to something "raw" (assuming you even had fonts to start with).
- vortex_ape 8y agoThe extreme cases (where there is no/incorrect mapping between the glyph and the character they represent) are a real pain! This mapping is stored as a ToUnicode map inside the PDF. In the past I've used OCR to handle such cases but I'm planning to create an experimental interface where anyone can modify the ToUnicode map. The challenge would be to make the modifications automated/user friendly.
- aidos 8y agoWicked. Sometimes the ToUnicode map is missing so we actually rebuild it ourselves from other information we find in the PDF.
- aasasd 8y agoIf PDF is just "characters at coordinates" then getting the data out seems to require all functions of an OCR engine outside of character recognition per se (namely, layout detection). And with botched fonts, you essentially need the full package. I so much want to see the day when PDF is dead like Flash.
- spaceandshit 8y agoAnd replaced with what?
- aasasd 8y agoWith Latex published to HTML. I'm curious to hear what widespread use-cases of PDF aren't covered by HTML with additions like MathML. Even HTML with fallback to SVG for complex sections would be a gigantic step forward from PDF.
- nabla9 8y agoHTML and PDF are used for different purposes. PDF is file format presenting fixed-layout documents application independent manner. You don't want to lose universal standard for that. HTML is markup for presenting documents application dependent manner.
- aasasd 8y agoThey certainly should be used for different purposes, but currently I don't see why PDF is necessary for all the papers. Why do they need fixed layout? Plenty of them are already published in both PDF and HTML, what's different about the rest? It's especially baffling in the case of computer science and programming papers when the contents are the same as in blogs. I have dozens of PDFs in my reading queue, for which I'll probably have to buy a tablet. Why can't I read the same columns of text and pictures on my electroink reader when I can do that with HTML? Who the hell knows.
- nabla9 8y agoPDF is for long term archival. There is no standard and widely recognized long term archival format for HTML pages (with all the extras). Web ARChive (WARC) provides method for bundling all the stuff in file in one file, but that's not enough. Plus the files will be quite large. You just don't know how your HTML and JavaScript renders 10 - 15 years from now. If you look old Web Archieve files you start to see how they become crap over time.
- aasasd 8y agoHTML is the format. You pack it with images, CSS and whatever else, and you have the distribution format. > Web ARChive (WARC) provides method for bundling all the stuff in file in one file, but that's not enough. Plus the files will be quite large. Not enough how? What is there that you need besides what the server hands to you, if that's what rendered in the first place? What magical compression methods do you have in PDF that are better than ZIP compression used in MAFF, for example? > You just don't know how your HTML and JavaScript renders 10 - 15 years from now. If you look old Web Archive files you start to see how they become crap over time. Have a static HTML version that's rendered the same in the future. You know, the same way that you have a static PDF standard. How do you render Javascript in PDFs in a standard way? You don't use Javascript, that's how. Javascript is not for publication of static semantic text, so you don't use Javascript for papers, it's a no-brainer.
- vortex_ape 8y agoAdobe released a way to attach data tables with PDFs. But I think it hasn't been adopted fully, since many organizations that release open data as PDFs don't tag the accompanying data tables. https://www.w3.org/TR/WCAG20-TECHS/PDF6.html https://www.w3.org/TR/WCAG20-TECHS/PDF6.html
- TeMPOraL 8y agoSounds like OCR is really the universal method. I'm guessing it shouldn't even be as hard as full-blown OCR, since you have access to fonts used, so you can render known characters as a reference and pretty much run a per-pixel matching on rendered PDF.
- aidos 8y agoI think it's probably Stockholm syndrome but I quite enjoying digging into PDFs. There are actually other ways of embedding the text where you have more information to go on. "Characters at coordinates" is a particularly rough hand to be dealt in a PDF (though it definitely happens).
- jimmy1 8y agoI have been recently writing a library in go to fill out AcroForms, and really, the PDF spec and syntax isn't really that bad or hard to understand. Most everything is a dictonary object. I've seen waaaay worse over-abused OOP monstrosities of a code base in C# and Java PDF 2.0 (ISO 32000-2) has been out for a while and supposedly it has eliminated a lot of the cruft from the spec. I just wish it was open like PDF 1.7
- svat 8y agoThe great thing about PDF, even the reason for its existence and adoption, is that a (valid) PDF file will look exactly the same — the same characters in the same fonts at exactly the same positions on every page — on any printer or display across the world, and across time. With HTML (what you propose) it is hard even to get something to look the same at different browser window widths, let alone different devices or different versions of browsers. Sometimes that's what you want (and when the visual appearance is not important, it may make sense to not use PDF), but I definitely wouldn't want to see PDF “dead”.
- aasasd 8y agoYeah, except the devices are different, so there's no point in trying to use the same page size on them, you only torture the reader. > With HTML (what you propose) it is hard even to get something to look the same at different browser window widths, let alone different devices or different versions of browsers. Only if you're trying to use some fancy layout, or if your idea of ‘the same’ is literal. Use a simple ‘text, image, text’ layout like it's the days of HTML 2.0, but with better formatting—and you'll have zero problems reformatting for different displays or reflowing the document into columns. Notice how all popular content sites adopted this layout in the main content column of their pages—and the pages work nicely on both desktop and mobile devices, and are captured fine with Pocket, Evernote and the like. If you're trying to use a fancy layout for a paper-like publication, the question is why the hell you're doing that.
- ezequiel-garzon 8y agoOnly if you're trying to use some fancy layout, or if your idea of ‘the same’ is literal. Actually, the plainest CSS-free HTML renders inexplicably small in some modern flagship phones... I mean the proprietary viewport meta tag, which is in the process of becoming CSS: https://www.w3.org/TR/css-device-adapt-1/ https://www.w3.org/TR/css-device-adapt-1/
- svat 8y agoYes, my idea of “the same” is “the same”. I understand what I believe to be your actual point: it would be nice if documents were more often published in a format that doesn't completely fix their layout and visual appearance. And I agree with that! When I'm reading something purely for its information, and don't care too much about the appearance, I too would like it if it weren't in a visually-fixed format. (That's what I said in the first comment tot: “when the visual appearance is not important, […] not use PDF”.) But my point is that for the goal of completely fixing the visual appearance, PDF is a pretty decent format (better than say, photographic images of the page), which is why it exists. When you say you want “the day when PDF is dead”, it appears as though you cannot imagine anyone wanting that goal. Here are two examples: Suppose you are an author of books (a physical artefact that will inhabit libraries for centuries; forget about digital displays and all that nonsense for a moment) and care about their typographic quality. Then you will want to make sure of things like: • that each paragraph contains appropriate line-breaks (http://eprg.org/G53DOC/pdfs/knuth-plass-breaking.pdf http://eprg.org/G53DOC/pdfs/knuth-plass-breaking.pdf), so that the page as a whole has a good “texture” or “greyness” (or “colour”), • that the words have hyphenation (to make the aforementioned good line-breaks possible), but not any poor hyphenation (https://tug.org/docs/liang/ https://tug.org/docs/liang/), • that the typefaces chosen are in harmony with each other, that the paper size leads to a good “form factor” for your book, and is appropriate given the kind of binding used, etc. • and finally, that after you have carefully proofread and verified every line of every page, the reader will not see something totally different, with lines of different widths broken in different places, etc. Or if you cannot relate to that example, then forget all that, because it's just a special case of a simpler, more general case: suppose you know that your document is ultimately going to be read on paper, and you'd like to make sure it can look the same ten years from now as it does today. Then PDF (especially PDF/A) is a decent format for this case. (PS: I've seen very few websites that have good typography in the sense that when printed they approach anything like the quality of a halfway decent book.)
- agumonkey 8y agoPDF (especially full spec'd PDF) seems bad, but I never like reading epub or other formats. PDFs are the static builds of documents.
- pwg 8y ago> all functions of an OCR engine outside of character recognition per se Actually, depending upon how 'obfuscated' an author was attempting to be, you might need that OCR engine itself. PDF allows for defining arbitrary mappings from byte values to font glyphs. So one could define byte value 32 (decimal, usually ASCII space) to actually map to printing, say, a capitol letter Z instead. One is supposed to provide a reverse mapping table when one does this that says "a decimal 32 byte prints a capitol letter Z" to allow for search and extraction purposes. But the PDF spec. does not require this reverse table be present. So it is quite possible to randomly assign font glyphs to arbitrary byte values, and omit the reverse mapping table. The result would be that extracting data back out of that PDF results in garbage if one does not know beforehand what the mapping from byte value to glyph was. So, if a 'bad actor' did this, one's only recourse to retrieving data would be to rasterize the PDF to a bitmap, then OCR the resulting bitmap to extract the content back out.
- tomrod 8y agoI've had this issue with LaTeX produced docs before (scans bad actor).
- nuclx 8y agoOr perform frequency analysis on the simple substitution chiffre. Seriously though, we need a document format with easier to extract payloads. Like Office documents with stronger structure, an underlying schema, along the lines of react-json-schema-form for Word.
- emj 8y agoODF - Open document format, it is not perfect but a lot better than Microsofts formats.
- maxxxxx 8y ago"I so much want to see the day when PDF is dead like Flash. " Totally agree. A while ago I had to write code to import a ton of PDF files and it was just infuriating to realize that we have data in highly structured documents, throw all structure away to create a PDF and then we somehow have to divine that structure back from the PDF with enormous effort and only partial success. It's just a horrible, horrible file format for what it's used now.