5 ms·

Sounds like this would massively simplify converting scientific articles back to HTML or text. Seriously, I know of one experiment that has O(thousands) of pap

by dguest 2y ago

Sounds like this would massively simplify converting scientific articles back to HTML or text. Seriously, I know of one experiment that has O(thousands) of papers as latex source, which they publish as PDFs and which (more recently) they convert back to markdown with some ML-based image to text engine. All so that it can be fed to a vector database to make searching easier, of course. PDFs are a terrible format for machine readable information (and thus for reproducible science), but they are the currency of the scientific community and as such will be the standard output for the foreseeable future.

nabla9 2y ago
>PDFs are a terrible format for machine readable information was terrible format. PDF 2.0 Tagged PDF has MathML support, Index, Quote, BibEntry, Title, lists, tables, .... you can derive HTML from PDF if it's fully tagged. PDF also support displaying and print production better than anything.
- elric 2y ago
  I'd argue it still is a terrible format for anything other than printing. It values form over function. I mean this makes sense for laying out papers to be published into a physical journal, but it doesn't make as much sense for other purposes. I'd much rather have plain text, or markdown, or asciidoc than PDF or HTML. It works everywhere. And support for embedded mathematical typesetting is a solved problem.
  - tjoff 2y ago
    I agree plain text has a lot of appeal. But support for embedded mathematical typesetting, images etc, being solved feels like a stretch. I mean sure, you can do it, but now you are on the other end of function over form. I feel like PDFs are, for tons of usecases, the best we've got. The alternative isn't markdown, it's usually .docx.
    - SSLy 2y ago
      The actual alternative is EPUBs
      - lxgr 2y ago
        I love ePub, but I also have to admit that the viewer experience for non-books is really not quite there yet. Most viewers on iOS, macOS, Linux and Android that I've tried are very tied to the idea of all ePubs being ebooks and insist on managing them in their own library, don't optimize latency to first page rendered (because they usually index and cache a bunch of stuff for a "newly opened book", assuming you'll be reading it over the course of days/weeks) etc. For example, on macOS, the default app associated with ePub is Apple Books, but to be able to use technical papers in ePub format, I really need something like Preview.app (and that doesn't support them). (In that way, the situation is very similar to JPEG XL: macOS supports them in quick view, but only Safari can actually open them...)
        cess11 2y ago
        Did you try muPDF?
        lxgr 2y ago
        Is that available for macOS? I can't seem to find a recent release.
        SSLy 2y ago
        https://formulae.brew.sh/formula/mupdf#default https://formulae.brew.sh/formula/mupdf#default
        lxgr 2y ago
        Thank you! Not sure how it is supposed to look like on other OSes, but this is barely a GUI application on macOS. (I have to launch it from the command line, UI and text rendering are extremely pixelated etc.) It's definitely not up to par with any PDF viewer I've used so far, especially when it comes to quickly navigating between pages and chapters (crucial for non-linearly working with scientific papers etc).
- lxgr 2y ago
  These features all sound great, but it's still extending PDF "upwards" from a quite low-level, representation-focused format. For most content PDFs are used today, I'd much prefer extending something like ePub "downwards" with stylesheets, rendering hints etc. – that can all just be thrown out by an accessibility- or readability-focused viewer. Yet, the PDF ecosystem is so large at this point, and so many organizations are still tied to skeuomorphic ideas focused around printed pages of text, that I can't see it happen anytime soon. I just hope that at least academia figures something out. ArXiv's recent HTML efforts [1] are a great step in that direction, but as I understand it, they still use PDF as their main ingestion format. [1] https://blog.arxiv.org/2023/12/21/accessibility-update-arxiv-now-offers-papers-in-html-format/ https://blog.arxiv.org/2023/12/21/accessibility-update-arxiv...

by dguest 2y ago

nabla9 2y ago

>PDFs are a terrible format for machine readable information was terrible format. PDF 2.0 Tagged PDF has MathML support, Index, Quote, BibEntry, Title, lists, tables, .... you can derive HTML from PDF if it's fully tagged. PDF also support displaying and print production better than anything.

elric 2y ago
I'd argue it still is a terrible format for anything other than printing. It values form over function. I mean this makes sense for laying out papers to be published into a physical journal, but it doesn't make as much sense for other purposes. I'd much rather have plain text, or markdown, or asciidoc than PDF or HTML. It works everywhere. And support for embedded mathematical typesetting is a solved problem.
- tjoff 2y ago
  I agree plain text has a lot of appeal. But support for embedded mathematical typesetting, images etc, being solved feels like a stretch. I mean sure, you can do it, but now you are on the other end of function over form. I feel like PDFs are, for tons of usecases, the best we've got. The alternative isn't markdown, it's usually .docx.
  - SSLy 2y ago
    The actual alternative is EPUBs
    - lxgr 2y ago
      I love ePub, but I also have to admit that the viewer experience for non-books is really not quite there yet. Most viewers on iOS, macOS, Linux and Android that I've tried are very tied to the idea of all ePubs being ebooks and insist on managing them in their own library, don't optimize latency to first page rendered (because they usually index and cache a bunch of stuff for a "newly opened book", assuming you'll be reading it over the course of days/weeks) etc. For example, on macOS, the default app associated with ePub is Apple Books, but to be able to use technical papers in ePub format, I really need something like Preview.app (and that doesn't support them). (In that way, the situation is very similar to JPEG XL: macOS supports them in quick view, but only Safari can actually open them...)
      - cess11 2y ago
        Did you try muPDF?
        lxgr 2y ago
        Is that available for macOS? I can't seem to find a recent release.
        SSLy 2y ago
        https://formulae.brew.sh/formula/mupdf#default https://formulae.brew.sh/formula/mupdf#default
        lxgr 2y ago
        Thank you! Not sure how it is supposed to look like on other OSes, but this is barely a GUI application on macOS. (I have to launch it from the command line, UI and text rendering are extremely pixelated etc.) It's definitely not up to par with any PDF viewer I've used so far, especially when it comes to quickly navigating between pages and chapters (crucial for non-linearly working with scientific papers etc).
lxgr 2y ago
These features all sound great, but it's still extending PDF "upwards" from a quite low-level, representation-focused format. For most content PDFs are used today, I'd much prefer extending something like ePub "downwards" with stylesheets, rendering hints etc. – that can all just be thrown out by an accessibility- or readability-focused viewer. Yet, the PDF ecosystem is so large at this point, and so many organizations are still tied to skeuomorphic ideas focused around printed pages of text, that I can't see it happen anytime soon. I just hope that at least academia figures something out. ArXiv's recent HTML efforts [1] are a great step in that direction, but as I understand it, they still use PDF as their main ingestion format. [1] https://blog.arxiv.org/2023/12/21/accessibility-update-arxiv-now-offers-papers-in-html-format/ https://blog.arxiv.org/2023/12/21/accessibility-update-arxiv...