7 ms·

I have a similar idea with PDF documents. Instead of having the royal PITA of parsing generated PDFs (e.g. invoices), things would be much simpler if every gene

I have a similar idea with PDF documents. Instead of having the royal PITA of parsing generated PDFs (e.g. invoices), things would be much simpler if every generated PDF came with a built-in SQLite or JSON that contains the structured data of that PDF. One day I will do it. Speaking more broadly, whether we talk about HTML or PDF it's the same problem: documents should have two representations - human-friendly and machine-friendly until AI gets so good that only having the human-friendly representation is enough.

ttyprintk 5y ago
If you want to start with the most controllable representation of a piece of paper, consider that OpenOffice (word processing and drawing) embeds its structured file format into PDFs. Maybe those are the PDFs we scrape first, leaving the bag-of-jpegs for later.
daveydave 5y ago
I have an application that converts word documents to RDF conformant with the SPAR ontologies (mainly DoCO http://www.sparontologies.net/ontologies/doco http://www.sparontologies.net/ontologies/doco), so it contains things like headers, numbering, contains/within relationships explicit in the RDF. I've used it successfully with PDFs by converting to DOCX first. Is this the sort of thing you had in mind? Not here to sell it! I think this is a genuinely interesting unexplored area ..
- dgudkov 5y ago
  The PDF format supports attachments (embedded files). I'm thinking about a set of libraries and/or a command-line utility that would make it trivially easy to attach a SQLite|JSON file to a PDF or extract one from a PDF. This won't fix existing files, of course, but at least for those apps that generate PDFs it will be easier to embed a SQLite/JSON into a generated PDF.
  - oever 5y ago
    XMP is meant for adding semantic information to (parts of) PDF files. https://en.wikipedia.org/wiki/Extensible_Metadata_Platform https://en.wikipedia.org/wiki/Extensible_Metadata_Platform
- jonnydubowsky 5y ago
  This looks awesome! The decision to combine structural and rhetorical ontologies, seems like it optimizes the best between cost and availability, in the sweetSpot of the users actual requirements when working with research and academic documents. Is this compatible with User Defined Language? https://ivan-radic.github.io/udl-documentation/ https://ivan-radic.github.io/udl-documentation/
  - daveydave 5y ago
    The RDF output I would typically serialise as turtle, which I believe there is existing UDL for in notepad++ though I don't use it
fatcow 5y ago
There's the EPC QR code standard in Europe which is underused... https://en.wikipedia.org/wiki/EPC_QR_code https://en.wikipedia.org/wiki/EPC_QR_code
__MatrixMan__ 5y ago
I have a similar, "One day I will do it" project. The idea is that somebody cares enough to make the semantic web work--it's just not the people with write access to the data. I think we can use CTPH algorithms to fingerprint the data independently of whatever names are used, and then we can use that to find representations of the "same data" submitted by other users. Probably there would be some reputation stuff involved, a web of trust, etc. The flow would go something like this: COGNIZE: 0. Encounter messy data in the wild (has pagination, timestamps of access, etc), need other representation (human/computer/whatever) 1. Calculate CTPH fingerprints, use them to search for link: miss 3. Clean data the hard way and publish canonical representation (ipfs?) 4. Generate missing representation the hard way, publish that too 5. Calculate the fingerprints common to both the missing representation and the canonical representation, and publish it as a "link" between the two. Unlike traditional web links, this one is bidirectional. RECOGNIZE 1. Different user encounters "same" data in the wild 2. Calculate fingerprints, use them to search for canonical representation: hit 3. Find further links to see what other representations of the "same data" are available, download them if desired. The fingerprint stuff works, but there's a lot of work left to be done re: mapping fuzzy hashes of "in the wild" data to cryptographic ones of "canonical representations" and finding ways to incentivize users to go through the hassle of the "cognize" step so that other users can benefit from the "recognize" step. Sorry for talking your ear off, it just feels good to know I'm not the only one working on something like this, even if our approaches are quite different (mine works on PDF's only because it works on arbitrary bytes). Good luck with yours.
- melony 5y ago
  Manual curation was why Berners-lee's Semantic Web failed. Unlike ARIA annotations which is required to serve certain clientele, the semantic web offers no inherent value to the developer. There is an entire field of machine learning dedicated to automatically generating knowledge graphs, I would start there rather than trying to manually curate and annotate.
  - __MatrixMan__ 5y ago
    That's a good point. Once the tooling is there for the manual workflow, I expect that an ML-driven approach will plug in to the same hooks without fuss. But I don't think it changes the main problem to be solved, which is that it's data consumers who want the semantic web, but so far it's been up to data providers to implement it. We need to be able to create links between data with without the participation of whoever hosts it.
visarga 5y ago
ML based invoice processing quality is around 90%, so human in the loop is still needed.
airstrike 5y ago
> Speaking more broadly, whether we talk about HTML or PDF it's the same problem: documents should have two representations Wait until you realize this applies to business presentations (Excel and PPT-authored-PDFs)
rokku 5y ago
in germany we have sth like ZUGFerD (in german it means carthorse). it is a pdf with a xml file for parsing.
biztos 5y ago
I like this, and of course you could also embed the text of the document. Nothing stops us from doing this right now. But: don't we need some way to prove that the data matches what is visually rendered in the PDF reader? And if we can prove that the embedded data matches the rendered document, couldn't that same logic just be used in reverse to generate the structured data from the renderable PDF?
- xmprt 5y ago
  That's not necessary. If you think of the PDF as a checksum then it's possible to have a one way function that generates the PDF (checksum) but that you can't retrieve the original JSON from. I do really like the idea of having a checksum of some sort if we end up embedding metadata like this.
  - q-base 5y ago
    That would be necessary. It should probably be baked into the creation of the file and perhaps even be included in the footer of the document itself.
  - biztos 5y ago
    That's a good idea: the tool that processes the data can just run your function and if the file doesn't match the result then it's rejected. But in the real world people are going to want to annotate the PDFs, there will be open-and-save cycles that add metadata and break the checksum, etc and so on, even without considering malicious actors. Restricting all that is maybe easy -- just reject anything that doesn't match the checksum, done -- but communicating that restriction to the users without making a mess of it is probably hard. Long ago I wrote some PDF generating programs and it was a lot of fun, but the spec has evolved in ways I imagine would make it less fun today. Still, could be a cool thing, and I'd be surprised if someone hasn't already done a version of it somewhere. [Edit: plus, whoever creates the one-way function is deciding what all the PDFs are going to look like, which means you will end up with many such functions to accommodate the different rendering goals, and then each validator needs to know each one and someone decides which ones to trust, and so on...]
DyslexicAtheist 5y ago
> ... documents should have two representations - human-friendly and machine-friendly until AI gets so good ... when I download papers from arxiv I sometimes choose the LaTex version because it often comes with commented out ideas that didn't make it into the paper. The author thought process becomes clear. The Metadata helps me understand the whole thing quicker the same way the semantic web helps the machine. Perhaps there is a clever philosophical analogy in there somewhere about "us becoming the machine" or "the map becoming the territory", but I can't put my finger on it.
- TuringTest 5y ago
  > Perhaps there is a clever philosophical analogy in there somewhere about "us becoming the machine" or "the map becoming the territory", but I can't put my finger on it. I personally believe the key is in "information architecture". We have been conveying information as a linear sequence of words for so long that we don't know yet how to best exploit non-linear formats. Programming languages harness the relation between specific instructions and the structure on which they are embedded; but this structure is oriented towards building a single executable block that controls a machine step by step. We have yet to build tools equivalent to IDEs to better exploit the overall structure of knowledge, but for the goal of understanding a topic at all levels of detail, bot at the local flow of ideas and the overall relation between its subtopics. The first widespread step in that direction was the 1.0 static World Wide Web, and we have learnt a lot thanks to it so we can now improve upon it. I have great hopes in online notebooks and no-code spreadsheet-like tools as the basis of such information-processing environments.
mpweiher 5y ago
The ZUGFeRD standard in Germany does this, though the embedded data is XML. https://de.wikipedia.org/wiki/ZUGFeRD https://de.wikipedia.org/wiki/ZUGFeRD My CodeDraw app also puts the source code in PDFs or PNGs that it generates.
manggit 5y ago
I think we need to reframe the problem. If we think of it as JSON data that comes with a PDF (similar to what @xmprt suggests "PDFs as checksum") then we have the benefit of machine-readable data that is transportable but also the attached human-readable PDF version of the data. This is exactly what we are trying to achieve at Anvil. 1. Provide the no-code tools to make it easy to convert existing PDF forms into web forms. 2. Share the web forms with perspective customers instead of PDF forms as email attachments 3. PDFs are generated as part of the workflow once the data is captured and represented in structured JSON. 4. (optional) request certification of the PDF via e-signatures The end result is a JSON payload that can be shared via API as well as a static PDF that is stored for human consumption. In most cases, we find that our customers actually just use the PDF as an interface with legacy systems (IRS, Banks, Insurance Companies) that haven't yet figured out how to modernize to a data-first business model. Of course this really only addresses PDFs that are used for information capture and transfer between two parties. But most PDFs that are not "standardized-forms" are made for consumption by humans not by machines (think ebooks, journal articles, graphics etc), and therefore having a JSON payload of the data attached doesn't really matter.

ttyprintk 5y ago

If you want to start with the most controllable representation of a piece of paper, consider that OpenOffice (word processing and drawing) embeds its structured file format into PDFs. Maybe those are the PDFs we scrape first, leaving the bag-of-jpegs for later.

daveydave 5y ago

I have an application that converts word documents to RDF conformant with the SPAR ontologies (mainly DoCO http://www.sparontologies.net/ontologies/doco http://www.sparontologies.net/ontologies/doco), so it contains things like headers, numbering, contains/within relationships explicit in the RDF. I've used it successfully with PDFs by converting to DOCX first. Is this the sort of thing you had in mind? Not here to sell it! I think this is a genuinely interesting unexplored area ..

dgudkov 5y ago
The PDF format supports attachments (embedded files). I'm thinking about a set of libraries and/or a command-line utility that would make it trivially easy to attach a SQLite|JSON file to a PDF or extract one from a PDF. This won't fix existing files, of course, but at least for those apps that generate PDFs it will be easier to embed a SQLite/JSON into a generated PDF.
- oever 5y ago
  XMP is meant for adding semantic information to (parts of) PDF files. https://en.wikipedia.org/wiki/Extensible_Metadata_Platform https://en.wikipedia.org/wiki/Extensible_Metadata_Platform
jonnydubowsky 5y ago
This looks awesome! The decision to combine structural and rhetorical ontologies, seems like it optimizes the best between cost and availability, in the sweetSpot of the users actual requirements when working with research and academic documents. Is this compatible with User Defined Language? https://ivan-radic.github.io/udl-documentation/ https://ivan-radic.github.io/udl-documentation/
- daveydave 5y ago
  The RDF output I would typically serialise as turtle, which I believe there is existing UDL for in notepad++ though I don't use it

fatcow 5y ago

There's the EPC QR code standard in Europe which is underused... https://en.wikipedia.org/wiki/EPC_QR_code https://en.wikipedia.org/wiki/EPC_QR_code

__MatrixMan__ 5y ago

I have a similar, "One day I will do it" project. The idea is that somebody cares enough to make the semantic web work--it's just not the people with write access to the data. I think we can use CTPH algorithms to fingerprint the data independently of whatever names are used, and then we can use that to find representations of the "same data" submitted by other users. Probably there would be some reputation stuff involved, a web of trust, etc. The flow would go something like this: COGNIZE: 0. Encounter messy data in the wild (has pagination, timestamps of access, etc), need other representation (human/computer/whatever) 1. Calculate CTPH fingerprints, use them to search for link: miss 3. Clean data the hard way and publish canonical representation (ipfs?) 4. Generate missing representation the hard way, publish that too 5. Calculate the fingerprints common to both the missing representation and the canonical representation, and publish it as a "link" between the two. Unlike traditional web links, this one is bidirectional. RECOGNIZE 1. Different user encounters "same" data in the wild 2. Calculate fingerprints, use them to search for canonical representation: hit 3. Find further links to see what other representations of the "same data" are available, download them if desired. The fingerprint stuff works, but there's a lot of work left to be done re: mapping fuzzy hashes of "in the wild" data to cryptographic ones of "canonical representations" and finding ways to incentivize users to go through the hassle of the "cognize" step so that other users can benefit from the "recognize" step. Sorry for talking your ear off, it just feels good to know I'm not the only one working on something like this, even if our approaches are quite different (mine works on PDF's only because it works on arbitrary bytes). Good luck with yours.

melony 5y ago
Manual curation was why Berners-lee's Semantic Web failed. Unlike ARIA annotations which is required to serve certain clientele, the semantic web offers no inherent value to the developer. There is an entire field of machine learning dedicated to automatically generating knowledge graphs, I would start there rather than trying to manually curate and annotate.
- __MatrixMan__ 5y ago
  That's a good point. Once the tooling is there for the manual workflow, I expect that an ML-driven approach will plug in to the same hooks without fuss. But I don't think it changes the main problem to be solved, which is that it's data consumers who want the semantic web, but so far it's been up to data providers to implement it. We need to be able to create links between data with without the participation of whoever hosts it.

visarga 5y ago

ML based invoice processing quality is around 90%, so human in the loop is still needed.

airstrike 5y ago

> Speaking more broadly, whether we talk about HTML or PDF it's the same problem: documents should have two representations Wait until you realize this applies to business presentations (Excel and PPT-authored-PDFs)

rokku 5y ago

in germany we have sth like ZUGFerD (in german it means carthorse). it is a pdf with a xml file for parsing.

biztos 5y ago

I like this, and of course you could also embed the text of the document. Nothing stops us from doing this right now. But: don't we need some way to prove that the data matches what is visually rendered in the PDF reader? And if we can prove that the embedded data matches the rendered document, couldn't that same logic just be used in reverse to generate the structured data from the renderable PDF?

xmprt 5y ago
That's not necessary. If you think of the PDF as a checksum then it's possible to have a one way function that generates the PDF (checksum) but that you can't retrieve the original JSON from. I do really like the idea of having a checksum of some sort if we end up embedding metadata like this.
- q-base 5y ago
  That would be necessary. It should probably be baked into the creation of the file and perhaps even be included in the footer of the document itself.
- biztos 5y ago
  That's a good idea: the tool that processes the data can just run your function and if the file doesn't match the result then it's rejected. But in the real world people are going to want to annotate the PDFs, there will be open-and-save cycles that add metadata and break the checksum, etc and so on, even without considering malicious actors. Restricting all that is maybe easy -- just reject anything that doesn't match the checksum, done -- but communicating that restriction to the users without making a mess of it is probably hard. Long ago I wrote some PDF generating programs and it was a lot of fun, but the spec has evolved in ways I imagine would make it less fun today. Still, could be a cool thing, and I'd be surprised if someone hasn't already done a version of it somewhere. [Edit: plus, whoever creates the one-way function is deciding what all the PDFs are going to look like, which means you will end up with many such functions to accommodate the different rendering goals, and then each validator needs to know each one and someone decides which ones to trust, and so on...]

DyslexicAtheist 5y ago

> ... documents should have two representations - human-friendly and machine-friendly until AI gets so good ... when I download papers from arxiv I sometimes choose the LaTex version because it often comes with commented out ideas that didn't make it into the paper. The author thought process becomes clear. The Metadata helps me understand the whole thing quicker the same way the semantic web helps the machine. Perhaps there is a clever philosophical analogy in there somewhere about "us becoming the machine" or "the map becoming the territory", but I can't put my finger on it.

TuringTest 5y ago
> Perhaps there is a clever philosophical analogy in there somewhere about "us becoming the machine" or "the map becoming the territory", but I can't put my finger on it. I personally believe the key is in "information architecture". We have been conveying information as a linear sequence of words for so long that we don't know yet how to best exploit non-linear formats. Programming languages harness the relation between specific instructions and the structure on which they are embedded; but this structure is oriented towards building a single executable block that controls a machine step by step. We have yet to build tools equivalent to IDEs to better exploit the overall structure of knowledge, but for the goal of understanding a topic at all levels of detail, bot at the local flow of ideas and the overall relation between its subtopics. The first widespread step in that direction was the 1.0 static World Wide Web, and we have learnt a lot thanks to it so we can now improve upon it. I have great hopes in online notebooks and no-code spreadsheet-like tools as the basis of such information-processing environments.

mpweiher 5y ago

The ZUGFeRD standard in Germany does this, though the embedded data is XML. https://de.wikipedia.org/wiki/ZUGFeRD https://de.wikipedia.org/wiki/ZUGFeRD My CodeDraw app also puts the source code in PDFs or PNGs that it generates.

manggit 5y ago

I think we need to reframe the problem. If we think of it as JSON data that comes with a PDF (similar to what @xmprt suggests "PDFs as checksum") then we have the benefit of machine-readable data that is transportable but also the attached human-readable PDF version of the data. This is exactly what we are trying to achieve at Anvil. 1. Provide the no-code tools to make it easy to convert existing PDF forms into web forms. 2. Share the web forms with perspective customers instead of PDF forms as email attachments 3. PDFs are generated as part of the workflow once the data is captured and represented in structured JSON. 4. (optional) request certification of the PDF via e-signatures The end result is a JSON payload that can be shared via API as well as a static PDF that is stored for human consumption. In most cases, we find that our customers actually just use the PDF as an interface with legacy systems (IRS, Banks, Insurance Companies) that haven't yet figured out how to modernize to a data-first business model. Of course this really only addresses PDFs that are used for information capture and transfer between two parties. But most PDFs that are not "standardized-forms" are made for consumption by humans not by machines (think ebooks, journal articles, graphics etc), and therefore having a JSON payload of the data attached doesn't really matter.