5 ms·

I really enjoy this article. It's nice to see one thats not just bashing the format as bad or insulting the developers. Did Microsoft solve any of these proble

I really enjoy this article. It's nice to see one thats not just bashing the format as bad or insulting the developers. Did Microsoft solve any of these problems in their "newer" file formats (IIRC its something like .docx instead of .doc)? And are those as crazy after a few years or have things gotten better since then?

yuhong 10y ago
I think it is better than the older binary formats, being based on XML.
gizmo686 10y ago
.docx is a much nicer format than doc. It is literally a zip file containing a some xml files [0]. Having said that, the spec is still 106 pages, but the entire xml schema is only 8 pages; the rest is exposistion. [0] Some features (such as embedding images) probably involve including non-xml files. [1] https://msdn.microsoft.com/en-us/library/dd773189(v=office.12).aspx https://msdn.microsoft.com/en-us/library/dd773189(v=office.1...
- userbinator 10y ago
  You only linked to the "extensions" MS added on top of OOXML, whose specs are in total an order of magnitude bigger. ISO/IEC-29500 or ECMA-376 is the actual spec, and its first part (of 5) is 5000+ pages. In comparison, 349 pages for the main spec + a 9 page auxillary spec for the document "filesystem" seems almost trivial. http://activityworkshop.net/rants/ooxml.html http://activityworkshop.net/rants/ooxml.html
- lqdc13 10y ago
  The PDF spec is 1000 pages. Nobody seems to complain. Or at least they complain a lot less.
  - tonyedgecombe 10y ago
    You can ignore about 800 pages of that for most applications though.
    - alexott 10y ago
      Not really, when you start to implement it, the more and more functionality required as you get more samples.
  - startling 10y ago
    Fortunately the only people who can read 1000-page documents are those who already have working PDF renderers, so the problem is self-limiting.
- frik 10y ago
  Much nicer? Not really. It literally a transpilation of the binary format. The XML files a very abnormal for everyone but who are familar how the original older doc/xls/ppt fileformat works. E.g. Look at how the Bold-tag is implemented, no sane developer would implement it that way. It's clearly done to mske it as straight forward port of the old format and make it as difficult as possible for every one else. The XML even contains binary blobs and all. A low-light. Every other modern Office format is nicer. https://en.wikipedia.org/wiki/Comparison_of_Office_Open_XML_and_OpenDocument https://en.wikipedia.org/wiki/Comparison_of_Office_Open_XML_... https://en.wikipedia.org/wiki/Standardization_of_Office_Open_XML#Criticism https://en.wikipedia.org/wiki/Standardization_of_Office_Open...
  - makomk 10y ago
    Worse, it's designed in a way that works completely differently from every other XML office format to allow Microsoft to patent it. Doing document markup the normal way in XML has too much prior art to be patentable (though that didn't stop Microsoft from trying), but they managed to get a patent on their oddball way of doing things.
userbinator 10y ago
And are those as crazy after a few years or have things gotten better since then? If anything, I'd say they're worse. https://en.wikipedia.org/wiki/Office_Open_XML https://en.wikipedia.org/wiki/Office_Open_XML XML'ized everything (yuck!), and instead of "just" Microsoft having control over the format, it's now a whole design-by-committee bureaucratically bloated mess with other parties trying to stuff things in too.
- chris_wot 10y ago
  XML is a perfectly reasonable format for documents.
  - userbinator 10y ago
    Perhaps for documents which are intended to be manually edited in a text editor, which is hardly the use-case here. Otherwise it's just extremely inefficient because of all the parsing involved. You can't know when one element ends or another begins, or skip through the parts you're not interested in, without reading every single byte and parsing it. Your XML parser has to handle extra whitespace (which will almost never be present), various quoting options, and arbitrary attributes/tags; all "decision points" within the code that add complexity. With a TLV format like the Excel binary one, you can seek through the records without knowing what they contain just by reading the header and adding its length field to the current offset. Then when you get to the record you want (identified by a simple integer comparison), just do a length check and read into the appropriate structure. There's far less to do, far less decision points, and far less to go wrong.
    - chris_wot 10y ago
      Except, of course, if malicious files are supplied or they are corrupted somehow.
      - userbinator 10y ago
        Regardless of XML-ness, you still have to check that the values are valid. With XML, you also have to check whether they're of the right type, because the format is so much more flexible.
thristian 10y ago
My understanding is that the .docx format is basically just the .doc format, transcribed to XML, and with a .zip filesystem instead of an OLE Compound Document filesystem. In fact, I recall some Office-competitor developers saying they preferred .doc, because (like Joel says) they could just read it into RAM and use it, instead of having to deserialize XML.
- 13of40 10y ago
  There are some parts of it, like VB macros, that are still stored in the old format before being put in the zip file.
- davidgerard 10y ago
  In practice there are assorted binary or encoded lumps. More fun: you know what happens if you save a .DOC in Office 2007/2010/2013? Any features past 2003 are saved as a lump of OOXML! http://vmiklos.hu/blog/doctok.html#comment-header http://vmiklos.hu/blog/doctok.html#comment-header
niftich 10y ago
Back in 2007 I was a proponent of the 'No-OOXML' movement [1], which outlined several objections (linked in the left margin on their page), some technical, some political, to the ISO standardization process of the new XML formats. In my opinion, the new formats are little more than XMLified representations of implementation details of how MS Office works; there is a distinct proliferation of elements that aren't purely semantic, and an awkward-but-inconsistent lack of separation of structure from presentation. But I'm not an expert on the binary formats nor the inner workings of their applications. Though my opinions remain unchanged after 9 years, in retrospect I do like that the new formats are documented by a true standards body, and are interoperable not just by reverse-engineering a closed format. [1] http://noooxml.wikidot.com/start http://noooxml.wikidot.com/start
- yuhong 10y ago
  I once suggested getting rid of the name and the ISO "standard" but keeping the file format itself in a poorly written blog article.
- 0x0 10y ago
  The best example on how this "just serialize our program's operations" style of file formats turned out crazy was the WMF exploit. Merely hovering over a malicious wmf file in explorer would trigger the payload. And the bug was literally a GDI method being played back which was "call my callback function". https://blogs.technet.microsoft.com/markrussinovich/2006/01/18/inside-the-wmf-backdoor/ https://blogs.technet.microsoft.com/markrussinovich/2006/01/...
- jordigh 10y ago
  But the old formats are also (mostly) documented by Microsoft. Only once did I have to reverse-engineer a magic number, and only to get something working with Excel 97: http://phpexcel.codeplex.com/discussions/71551 http://phpexcel.codeplex.com/discussions/71551 For the most part, with Word I get better compatibility in LO with the old binary formats than with OOXML. If someone sends me .docx and it looks garbled, it's easier to ask them to resave as .doc than to try to get them to produce .odt
narag 10y ago
It's nice to see one thats not just bashing the format as bad or insulting the developers. You know the author was part of Excel team, don't you? The article seems like an explanation, not a justification. Moreover, I totally agree with the conclusion: use OLE to extract information programatically. But the format is still terrible :-) The fact that there were reasons to do it that way at the time is not the same as saying the decisions were the right ones. Probably there were reasons to do <insert here any despicable historical or technical catastrophe>.
- youdontknowtho 10y ago
  "the right ones"...what would the "rights ones" have been besides shipping functioning software for their users and making money for their shareholders?
  - narag 10y ago
    Functioning and right are not the same thing. And right for the shareholders is a strange way of defining what's right. Strange for me anyway.
    - youdontknowtho 10y ago
      That's kind of my point. On this forum we tend to define "right" by a certain standard of engineering aesthetics. I say a certain standard of aesthetics because it definitely isn't the only one. Other definitions of "right" could reasonably include "our software performed the intended functions expected by the people that paid for it." One common definition of "right" in capitalist countries is the fiduciary responsibility to maximize profit. It's not one that I personally think works in every scenario. That version of "right" can often find it's self in direct contradiction of definitions from moral, social, and legal areas. My only point was that "right" means a very particular thing on HN. It means Unix-y...but not actually Unix...its this Unix that inhabits someone's mind. Where X and sockets are also "files" and text is the only RPC system you would ever need. It's also functional programming fetishistic. LISP because of the great PG. Erlang because I heard of this one really awesome app with great up time...and have you heard about Plan 9, it's perfect if it weren't for {insert dark force of mediocracy} that kept it from being a winner. Thanks for asking. I've been needing to say that to someone for a long time.
jesserosenthal 10y ago
I wrote an open-source reader for docx, including for math equations, in a week or two of part-time hobby work. The spec isn't always sensible (though I'm sure there are historical reasons for most of the decisions) but it's always easy enough to look at the xml and figure it out. Most importantly, your language only requires a zip lib and an xml parsing lib. So, yeah, I'd say it's much nicer to deal with than what this article describes.

yuhong 10y ago

I think it is better than the older binary formats, being based on XML.

gizmo686 10y ago

.docx is a much nicer format than doc. It is literally a zip file containing a some xml files [0]. Having said that, the spec is still 106 pages, but the entire xml schema is only 8 pages; the rest is exposistion. [0] Some features (such as embedding images) probably involve including non-xml files. [1] https://msdn.microsoft.com/en-us/library/dd773189(v=office.12).aspx https://msdn.microsoft.com/en-us/library/dd773189(v=office.1...

userbinator 10y ago
You only linked to the "extensions" MS added on top of OOXML, whose specs are in total an order of magnitude bigger. ISO/IEC-29500 or ECMA-376 is the actual spec, and its first part (of 5) is 5000+ pages. In comparison, 349 pages for the main spec + a 9 page auxillary spec for the document "filesystem" seems almost trivial. http://activityworkshop.net/rants/ooxml.html http://activityworkshop.net/rants/ooxml.html
lqdc13 10y ago
The PDF spec is 1000 pages. Nobody seems to complain. Or at least they complain a lot less.
- tonyedgecombe 10y ago
  You can ignore about 800 pages of that for most applications though.
  - alexott 10y ago
    Not really, when you start to implement it, the more and more functionality required as you get more samples.
- startling 10y ago
  Fortunately the only people who can read 1000-page documents are those who already have working PDF renderers, so the problem is self-limiting.
frik 10y ago
Much nicer? Not really. It literally a transpilation of the binary format. The XML files a very abnormal for everyone but who are familar how the original older doc/xls/ppt fileformat works. E.g. Look at how the Bold-tag is implemented, no sane developer would implement it that way. It's clearly done to mske it as straight forward port of the old format and make it as difficult as possible for every one else. The XML even contains binary blobs and all. A low-light. Every other modern Office format is nicer. https://en.wikipedia.org/wiki/Comparison_of_Office_Open_XML_and_OpenDocument https://en.wikipedia.org/wiki/Comparison_of_Office_Open_XML_... https://en.wikipedia.org/wiki/Standardization_of_Office_Open_XML#Criticism https://en.wikipedia.org/wiki/Standardization_of_Office_Open...
- makomk 10y ago
  Worse, it's designed in a way that works completely differently from every other XML office format to allow Microsoft to patent it. Doing document markup the normal way in XML has too much prior art to be patentable (though that didn't stop Microsoft from trying), but they managed to get a patent on their oddball way of doing things.

userbinator 10y ago

And are those as crazy after a few years or have things gotten better since then? If anything, I'd say they're worse. https://en.wikipedia.org/wiki/Office_Open_XML https://en.wikipedia.org/wiki/Office_Open_XML XML'ized everything (yuck!), and instead of "just" Microsoft having control over the format, it's now a whole design-by-committee bureaucratically bloated mess with other parties trying to stuff things in too.

chris_wot 10y ago
XML is a perfectly reasonable format for documents.
- userbinator 10y ago
  Perhaps for documents which are intended to be manually edited in a text editor, which is hardly the use-case here. Otherwise it's just extremely inefficient because of all the parsing involved. You can't know when one element ends or another begins, or skip through the parts you're not interested in, without reading every single byte and parsing it. Your XML parser has to handle extra whitespace (which will almost never be present), various quoting options, and arbitrary attributes/tags; all "decision points" within the code that add complexity. With a TLV format like the Excel binary one, you can seek through the records without knowing what they contain just by reading the header and adding its length field to the current offset. Then when you get to the record you want (identified by a simple integer comparison), just do a length check and read into the appropriate structure. There's far less to do, far less decision points, and far less to go wrong.
  - chris_wot 10y ago
    Except, of course, if malicious files are supplied or they are corrupted somehow.
    - userbinator 10y ago
      Regardless of XML-ness, you still have to check that the values are valid. With XML, you also have to check whether they're of the right type, because the format is so much more flexible.

thristian 10y ago

My understanding is that the .docx format is basically just the .doc format, transcribed to XML, and with a .zip filesystem instead of an OLE Compound Document filesystem. In fact, I recall some Office-competitor developers saying they preferred .doc, because (like Joel says) they could just read it into RAM and use it, instead of having to deserialize XML.

13of40 10y ago
There are some parts of it, like VB macros, that are still stored in the old format before being put in the zip file.
davidgerard 10y ago
In practice there are assorted binary or encoded lumps. More fun: you know what happens if you save a .DOC in Office 2007/2010/2013? Any features past 2003 are saved as a lump of OOXML! http://vmiklos.hu/blog/doctok.html#comment-header http://vmiklos.hu/blog/doctok.html#comment-header

niftich 10y ago

Back in 2007 I was a proponent of the 'No-OOXML' movement [1], which outlined several objections (linked in the left margin on their page), some technical, some political, to the ISO standardization process of the new XML formats. In my opinion, the new formats are little more than XMLified representations of implementation details of how MS Office works; there is a distinct proliferation of elements that aren't purely semantic, and an awkward-but-inconsistent lack of separation of structure from presentation. But I'm not an expert on the binary formats nor the inner workings of their applications. Though my opinions remain unchanged after 9 years, in retrospect I do like that the new formats are documented by a true standards body, and are interoperable not just by reverse-engineering a closed format. [1] http://noooxml.wikidot.com/start http://noooxml.wikidot.com/start

yuhong 10y ago
I once suggested getting rid of the name and the ISO "standard" but keeping the file format itself in a poorly written blog article.
0x0 10y ago
The best example on how this "just serialize our program's operations" style of file formats turned out crazy was the WMF exploit. Merely hovering over a malicious wmf file in explorer would trigger the payload. And the bug was literally a GDI method being played back which was "call my callback function". https://blogs.technet.microsoft.com/markrussinovich/2006/01/18/inside-the-wmf-backdoor/ https://blogs.technet.microsoft.com/markrussinovich/2006/01/...
jordigh 10y ago
But the old formats are also (mostly) documented by Microsoft. Only once did I have to reverse-engineer a magic number, and only to get something working with Excel 97: http://phpexcel.codeplex.com/discussions/71551 http://phpexcel.codeplex.com/discussions/71551 For the most part, with Word I get better compatibility in LO with the old binary formats than with OOXML. If someone sends me .docx and it looks garbled, it's easier to ask them to resave as .doc than to try to get them to produce .odt

narag 10y ago

It's nice to see one thats not just bashing the format as bad or insulting the developers. You know the author was part of Excel team, don't you? The article seems like an explanation, not a justification. Moreover, I totally agree with the conclusion: use OLE to extract information programatically. But the format is still terrible :-) The fact that there were reasons to do it that way at the time is not the same as saying the decisions were the right ones. Probably there were reasons to do <insert here any despicable historical or technical catastrophe>.

youdontknowtho 10y ago
"the right ones"...what would the "rights ones" have been besides shipping functioning software for their users and making money for their shareholders?
- narag 10y ago
  Functioning and right are not the same thing. And right for the shareholders is a strange way of defining what's right. Strange for me anyway.
  - youdontknowtho 10y ago
    That's kind of my point. On this forum we tend to define "right" by a certain standard of engineering aesthetics. I say a certain standard of aesthetics because it definitely isn't the only one. Other definitions of "right" could reasonably include "our software performed the intended functions expected by the people that paid for it." One common definition of "right" in capitalist countries is the fiduciary responsibility to maximize profit. It's not one that I personally think works in every scenario. That version of "right" can often find it's self in direct contradiction of definitions from moral, social, and legal areas. My only point was that "right" means a very particular thing on HN. It means Unix-y...but not actually Unix...its this Unix that inhabits someone's mind. Where X and sockets are also "files" and text is the only RPC system you would ever need. It's also functional programming fetishistic. LISP because of the great PG. Erlang because I heard of this one really awesome app with great up time...and have you heard about Plan 9, it's perfect if it weren't for {insert dark force of mediocracy} that kept it from being a winner. Thanks for asking. I've been needing to say that to someone for a long time.

jesserosenthal 10y ago

I wrote an open-source reader for docx, including for math equations, in a week or two of part-time hobby work. The spec isn't always sensible (though I'm sure there are historical reasons for most of the decisions) but it's always easy enough to look at the xml and figure it out. Most importantly, your language only requires a zip lib and an xml parsing lib. So, yeah, I'd say it's much nicer to deal with than what this article describes.