4 ms·
I'm curious, we use many PDF parsing and formatting tools as part of an ML ingest pipeline. Our goal is to keep the document as close as possible to the origina
by ddtaylor 2y ago
I'm curious, we use many PDF parsing and formatting tools as part of an ML ingest pipeline. Our goal is to keep the document as close as possible to the original in meaning, but remove unwanted junk and simplify the document by removing everything non-content related or converting it to text the ML can work with.
Surely you can do that instead? Parse the PDFs and format them in basic ways without support for "extensions" or anything. Let the user read that before using the "real" document with extensions potentially enabled.
- fl0id 2y agothen you at least need a pdf reader that implements that, and have to be sure that the parsing you do have cannot be exploited, while still giving a useful representation. This might be easier for ML where you don't care about visual display, but a human generally doesn't want to read raw, unformatted text. And a surprising amount of stuff is probably needed for a half-way decent visual display.
- ddtaylor 2y agoI do view the documents we format after they have gone through the processing stage. They seem to be the same in most ways I would care about. Diagrams are still present, etc. I don't know about PDFs that contain forms as these are not those kind of documents but closer to research documents.