6 ms·

It actually seems like a sane choice to me. PDF is good for rendering, but horrible for parsing. DOCX is a ZIP file with XML data. Maybe ODT or whatever would'v

It actually seems like a sane choice to me. PDF is good for rendering, but horrible for parsing. DOCX is a ZIP file with XML data. Maybe ODT or whatever would've been a better choice, I don't know what the format is like. But if you disregard the usual knee-jerk "but it's Microsoft!" reaction, it doesn't seem like a bad choice.

jedberg 4y ago
I never said it was a bad choice, only that they didn't say why they chose it.
- dataflow 4y ago
  I guess, but you were replying to "I was wondering why docx format would be chosen instead of PDF" seemingly unconvinced, so I assumed you thought PDF would've made more sense.
molsongolden 4y ago
Was just about to post this. Unzipping DOCX and parsing XML is much easier than accurately processing PDF submissions.
- oneplane 4y ago
  OCR has come a long way, so much that visually interpreting a PDF is about as error-prom as parsing XML output from Microsoft in non-microsoft software.
  - programmarchy 4y ago
    Try extracting tabular data from a PDF! With XML it's trivial, but for PDF you need highly specialized software packages to do this. One of the best, pdfplumber, is largely based [1] on a Master's thesis titled Algorithmic Extraction of Data in Tables in PDF Documents [2]. [1] https://github.com/jsvine/pdfplumber/blob/stable/pdfplumber/table.py#L551 https://github.com/jsvine/pdfplumber/blob/stable/pdfplumber/... [2] https://trepo.tuni.fi/bitstream/handle/123456789/21520/Nurminen.pdf https://trepo.tuni.fi/bitstream/handle/123456789/21520/Nurmi...
    - oneplane 4y ago
      This was mostly aimed at the various ways the XML document may or may not conform to any number of XSD types. What we 'see' as a table might not be described and stored as a table in the same way in XML. And with XML I mean whatever XML office (the one from microsoft) generates. A 6000-page spec and attributes that specify if the data is tabular data based on various properties (be it columns and rows or just plain text with start and stop pointers) and then may or may not render it visually as a table is error prone, even on first-party implementations (first-party desktop versions within Windows vary, as wel as on macOS, Android, iOS and their web offering). If there was one simple data structure describing the table and all other aspects being optional, then yes, a XML based format is easier than OCR. But that's not the case I was pointing at.
    - a1369209993 4y ago
      > for PDF you need highly specialized software packages to do this. Not really, or at least not all that specialized. You need: a: a pdf-to-raster-image converter (ie any working PDF viewer, plus maybe the X server it talks to) b: a reasonably decent OCR system capable of scanning tables (definitely nontrivial, but hardly "highly specialized" since things other than PDFs display data in tables).
- 0x457 4y ago
  Why not make yet another XML standard for patents and generate PDFs from it?
  - moistly 4y ago
    Or just use Docbook and the standard, open pipelines for publication to print and web.
    - 0x457 4y ago
      I was saying it half-sarcastically because of how many "universal" XML standards are out there and how useless they are. I wrote DDEX parsers in the past and I wish it never existed...
ndiddy 4y ago
The Office Open XML file format is extremely complex, and takes up around 6,500 pages (compared to ~1000 for ODF). One thing you notice when reading the DOCX spec is that they designed it with the sole constraint that DOC files could easily be converted to DOCX. For example, you'll frequently see compatibility tags like "autoSpaceLikeWord95", "footnoteLayoutLikeWW8", "useWord2002TableStyleRules", and "lineWrapLikeWord6" that expose internal implementation details. Rather than creating a useful standard allowing all users to store their documents in a clean, portable way, Microsoft decided to make their standard faithfully reproduce all of the quirks and bugs of their legacy binary formats. It's so difficult to correctly implement the Office Open XML standard that even Microsoft took until Office 2013 to do so (the standard was approved in 2006).
- dataflow 4y ago
  Interesting! How do they compare feature-wise? I feel like there must be things each of them support that the other one doesn't, but I don't know how consequential they are.
  - Kye 4y ago
    I don't know how thorough or accurate it is, but Microsoft has a list. https://support.microsoft.com/en-us/office/differences-between-the-opendocument-text-odt-format-and-the-word-docx-format-d9d51a92-56d1-4794-8b68-5efb57aebfdc https://support.microsoft.com/en-us/office/differences-betwe... The list was only a few lines the last time I looked years ago, so maybe they're actually trying to make a complete list.
- jfk13 4y ago
  So true. "It's XML, so it must easy to parse and manipulate" is such a naive, even misleading attitude. If what you do is take a byzantine, legacy-encrusted implementation and just serialise its data strucures to an XML representation, very little has been gained. [edit: but I will grant that almost anything is better than attempting to parse useful content from PDF.]
  - sidewndr46 4y ago
fezfight 4y ago
I think the knee-jerk is against any alternative to Office, not against Office. Statistically speaking, trying to use anything reasonable that doesn't genuflect to Microsoft's monopoly is what seems to be met with a knee-jerk reply such as yours. As in, there's probably more people who don't care but hate the complaining about libre stuff than their are advocates for libre stuff.
stefan_ 4y ago
Neither of these are sane. PDF and DOCX are both built to ape analog paper processes. It's been 40 years of this computer thing, it's time to get rid of the 70s office facsimiles.

jedberg 4y ago

I never said it was a bad choice, only that they didn't say why they chose it.

dataflow 4y ago
I guess, but you were replying to "I was wondering why docx format would be chosen instead of PDF" seemingly unconvinced, so I assumed you thought PDF would've made more sense.

molsongolden 4y ago

Was just about to post this. Unzipping DOCX and parsing XML is much easier than accurately processing PDF submissions.

oneplane 4y ago
OCR has come a long way, so much that visually interpreting a PDF is about as error-prom as parsing XML output from Microsoft in non-microsoft software.
- programmarchy 4y ago
  Try extracting tabular data from a PDF! With XML it's trivial, but for PDF you need highly specialized software packages to do this. One of the best, pdfplumber, is largely based [1] on a Master's thesis titled Algorithmic Extraction of Data in Tables in PDF Documents [2]. [1] https://github.com/jsvine/pdfplumber/blob/stable/pdfplumber/table.py#L551 https://github.com/jsvine/pdfplumber/blob/stable/pdfplumber/... [2] https://trepo.tuni.fi/bitstream/handle/123456789/21520/Nurminen.pdf https://trepo.tuni.fi/bitstream/handle/123456789/21520/Nurmi...
  - oneplane 4y ago
    This was mostly aimed at the various ways the XML document may or may not conform to any number of XSD types. What we 'see' as a table might not be described and stored as a table in the same way in XML. And with XML I mean whatever XML office (the one from microsoft) generates. A 6000-page spec and attributes that specify if the data is tabular data based on various properties (be it columns and rows or just plain text with start and stop pointers) and then may or may not render it visually as a table is error prone, even on first-party implementations (first-party desktop versions within Windows vary, as wel as on macOS, Android, iOS and their web offering). If there was one simple data structure describing the table and all other aspects being optional, then yes, a XML based format is easier than OCR. But that's not the case I was pointing at.
  - a1369209993 4y ago
    > for PDF you need highly specialized software packages to do this. Not really, or at least not all that specialized. You need: a: a pdf-to-raster-image converter (ie any working PDF viewer, plus maybe the X server it talks to) b: a reasonably decent OCR system capable of scanning tables (definitely nontrivial, but hardly "highly specialized" since things other than PDFs display data in tables).
0x457 4y ago
Why not make yet another XML standard for patents and generate PDFs from it?
- moistly 4y ago
  Or just use Docbook and the standard, open pipelines for publication to print and web.
  - 0x457 4y ago
    I was saying it half-sarcastically because of how many "universal" XML standards are out there and how useless they are. I wrote DDEX parsers in the past and I wish it never existed...

ndiddy 4y ago

The Office Open XML file format is extremely complex, and takes up around 6,500 pages (compared to ~1000 for ODF). One thing you notice when reading the DOCX spec is that they designed it with the sole constraint that DOC files could easily be converted to DOCX. For example, you'll frequently see compatibility tags like "autoSpaceLikeWord95", "footnoteLayoutLikeWW8", "useWord2002TableStyleRules", and "lineWrapLikeWord6" that expose internal implementation details. Rather than creating a useful standard allowing all users to store their documents in a clean, portable way, Microsoft decided to make their standard faithfully reproduce all of the quirks and bugs of their legacy binary formats. It's so difficult to correctly implement the Office Open XML standard that even Microsoft took until Office 2013 to do so (the standard was approved in 2006).

dataflow 4y ago
Interesting! How do they compare feature-wise? I feel like there must be things each of them support that the other one doesn't, but I don't know how consequential they are.
- Kye 4y ago
  I don't know how thorough or accurate it is, but Microsoft has a list. https://support.microsoft.com/en-us/office/differences-between-the-opendocument-text-odt-format-and-the-word-docx-format-d9d51a92-56d1-4794-8b68-5efb57aebfdc https://support.microsoft.com/en-us/office/differences-betwe... The list was only a few lines the last time I looked years ago, so maybe they're actually trying to make a complete list.
jfk13 4y ago
So true. "It's XML, so it must easy to parse and manipulate" is such a naive, even misleading attitude. If what you do is take a byzantine, legacy-encrusted implementation and just serialise its data strucures to an XML representation, very little has been gained. [edit: but I will grant that almost anything is better than attempting to parse useful content from PDF.]
- sidewndr46 4y ago

fezfight 4y ago

I think the knee-jerk is against any alternative to Office, not against Office. Statistically speaking, trying to use anything reasonable that doesn't genuflect to Microsoft's monopoly is what seems to be met with a knee-jerk reply such as yours. As in, there's probably more people who don't care but hate the complaining about libre stuff than their are advocates for libre stuff.

stefan_ 4y ago

Neither of these are sane. PDF and DOCX are both built to ape analog paper processes. It's been 40 years of this computer thing, it's time to get rid of the 70s office facsimiles.