4 ms·

Let's not underestimate the impact of such tool: we are talking about freeing up tons of knowledge from a "good for consumption/bad for distribution" format. I

Let's not underestimate the impact of such tool: we are talking about freeing up tons of knowledge from a "good for consumption/bad for distribution" format. I'm very excited about it. Let's build a pipeline: all the pdfs -> markdown them all -> archive.org them all

samuell 3y ago
Yes, there is an enormous interest in this kind of thing, not the least in larger organizations with tons of PDF documents in various forms. Even though this would only cover a small part of the needs or use cases, it will still be hugely useful if it works well.
- mannycalavera42 3y ago
  cough L cough L cough M cough anyone? :)
  - samuell 3y ago
    Yeah, I know, but a lot of this content can be pretty sensitive, and might not be allowed to upload outside organization networks sometimes (hospitals, governments etc).
    - scoot 3y ago
      Like most software, LLMs can be run locally, or on private infrastructure. This was on the front page yesterday, which is not the only way to run an LLM locally, but about the easiest way possible: https://news.ycombinator.com/item?id=38464057 https://news.ycombinator.com/item?id=38464057
      - samuell 3y ago
        Thanks! Well, yea, I just thought the quality of offline models might not yet be good enough. By I'm glad to be told otherwise :)
miki123211 3y ago
This also has tons of use-cases for accessibility, getting PDF accessibility right is tons of work and even if you manage it, it's highly likely that the PDF viewers your users use don't support the necessary standards anyway.
deleted 3y ago
[deleted]
Alex3917 3y ago
> we are talking about freeing up tons of knowledge from a "good for consumption/bad for distribution" format. FWIW PDF is actually great for distribution. It allows you to invisibly embed all the raw data used to generate the document that the end user is seeing, in whatever format you want. So if you are generating your PDFs by using PrinceXML to render HTML, you can embed the raw JSON used to generate all of the text, graphs, charts, etc. Now most people don't actually do this of course, but that isn't the fault of the spec.
- sertbdfgbnfgsd 3y ago
  pdfs don't play well with ereaders.
  - Alex3917 3y ago
    Are the standards for building accessible PDFs worse than the standards for building accessible websites, or are they just not as commonly implemented?
    - sertbdfgbnfgsd 3y ago
      I don't know anything about websites. I had ebooks in mind.
    - solardev 3y ago
      (anecdotally) PDFs usually come from many people, departments, companies, and apps. It's hard to shoehorn in accessibility if someone didn't add it in at the origin (like in indesign or whatever app they used). Or if they printed to PDF, whatever accessibility they had would probably be lost. Much of the time it's like working with a raster image with some embedded text. Not really the same as being able to edit a proper semantic document. With a website and available source code, any dev working on it later on can still add accessibility, tweak contrasts and fonts and add screen reader hints, etc. It's much harder to do so for PDFs after the fact. And PDF viewer apps may or may not even support the accessibility annotations. By contrast all the major browsers and operating systems have OK support for web accessibility.
vikp 3y ago
Author here - this is one of the reasons I made this. Also see https://github.com/VikParuchuri/libgen_to_txt https://github.com/VikParuchuri/libgen_to_txt , although I haven't integrated marker with it yet (it uses naive text extraction).
kevincox 3y ago
> Let's build a pipeline I don't think that is the right approach for archiving. The preferred pipeline would be all the pdfs -> archive them all -> markdown them This way you can always re-run the conversion as bugs are fixed and improvements are made. Generally archivist prefer to save as close to the source material as possible, because every transformation from there can only lose data.
- crotchfire 3y ago
  Yeah if you get down into the weeds these models are significantly corrupting the source data. I opened the first example to a random chapter (1.4 Formal and natural languages); within the first three paragraphs it: - Hallucinated spurious paragraph breaks - Ignored all the boldfacing - Hallucinated a blockquote into a new section This is not a tool to produce something for humans to read. Maybe it might be useful as part of some pipeline that needs to feed markdown into some other machine process. I would not waste my time reading the crud that came out of this thing. It's a stunt.
Gabrys1 3y ago
Finally a good usecase for AI/ML/LLM.

samuell 3y ago

Yes, there is an enormous interest in this kind of thing, not the least in larger organizations with tons of PDF documents in various forms. Even though this would only cover a small part of the needs or use cases, it will still be hugely useful if it works well.

mannycalavera42 3y ago
cough L cough L cough M cough anyone? :)
- samuell 3y ago
  Yeah, I know, but a lot of this content can be pretty sensitive, and might not be allowed to upload outside organization networks sometimes (hospitals, governments etc).
  - scoot 3y ago
    Like most software, LLMs can be run locally, or on private infrastructure. This was on the front page yesterday, which is not the only way to run an LLM locally, but about the easiest way possible: https://news.ycombinator.com/item?id=38464057 https://news.ycombinator.com/item?id=38464057
    - samuell 3y ago
      Thanks! Well, yea, I just thought the quality of offline models might not yet be good enough. By I'm glad to be told otherwise :)

miki123211 3y ago

This also has tons of use-cases for accessibility, getting PDF accessibility right is tons of work and even if you manage it, it's highly likely that the PDF viewers your users use don't support the necessary standards anyway.

deleted 3y ago

[deleted]

Alex3917 3y ago

> we are talking about freeing up tons of knowledge from a "good for consumption/bad for distribution" format. FWIW PDF is actually great for distribution. It allows you to invisibly embed all the raw data used to generate the document that the end user is seeing, in whatever format you want. So if you are generating your PDFs by using PrinceXML to render HTML, you can embed the raw JSON used to generate all of the text, graphs, charts, etc. Now most people don't actually do this of course, but that isn't the fault of the spec.

sertbdfgbnfgsd 3y ago
pdfs don't play well with ereaders.
- Alex3917 3y ago
  Are the standards for building accessible PDFs worse than the standards for building accessible websites, or are they just not as commonly implemented?
  - sertbdfgbnfgsd 3y ago
    I don't know anything about websites. I had ebooks in mind.
  - solardev 3y ago
    (anecdotally) PDFs usually come from many people, departments, companies, and apps. It's hard to shoehorn in accessibility if someone didn't add it in at the origin (like in indesign or whatever app they used). Or if they printed to PDF, whatever accessibility they had would probably be lost. Much of the time it's like working with a raster image with some embedded text. Not really the same as being able to edit a proper semantic document. With a website and available source code, any dev working on it later on can still add accessibility, tweak contrasts and fonts and add screen reader hints, etc. It's much harder to do so for PDFs after the fact. And PDF viewer apps may or may not even support the accessibility annotations. By contrast all the major browsers and operating systems have OK support for web accessibility.

vikp 3y ago

Author here - this is one of the reasons I made this. Also see https://github.com/VikParuchuri/libgen_to_txt https://github.com/VikParuchuri/libgen_to_txt , although I haven't integrated marker with it yet (it uses naive text extraction).

kevincox 3y ago

> Let's build a pipeline I don't think that is the right approach for archiving. The preferred pipeline would be all the pdfs -> archive them all -> markdown them This way you can always re-run the conversion as bugs are fixed and improvements are made. Generally archivist prefer to save as close to the source material as possible, because every transformation from there can only lose data.

crotchfire 3y ago
Yeah if you get down into the weeds these models are significantly corrupting the source data. I opened the first example to a random chapter (1.4 Formal and natural languages); within the first three paragraphs it: - Hallucinated spurious paragraph breaks - Ignored all the boldfacing - Hallucinated a blockquote into a new section This is not a tool to produce something for humans to read. Maybe it might be useful as part of some pipeline that needs to feed markdown into some other machine process. I would not waste my time reading the crud that came out of this thing. It's a stunt.

Gabrys1 3y ago

Finally a good usecase for AI/ML/LLM.