6 ms·

That would be coming full circle, and going back to the bad old days of binary .doc and .xls files, which used an efficient data store for fast incremental upda

That would be coming full circle, and going back to the bad old days of binary .doc and .xls files, which used an efficient data store for fast incremental updates: https://www.joelonsoftware.com/2008/02/19/why-are-the-microsoft-office-file-formats-so-complicated-and-some-workarounds/ https://www.joelonsoftware.com/2008/02/19/why-are-the-micros... OpenDocument and its competitor (OOXML) both use zip/XML for portability reasons. Yes, they're less efficient. But they're also completely independent of any specific software implementation [1]. [1] Mostly - OOXML has some parts that depend on MS Office's implementation. But ODF is completely described by the spec.

abtinf 9y ago
I don't see how your comment is responsive to the excellent article. doc and xls were proprietary binary formats. sqlite files couldn't be more open - the software to interact with them is in the public domain and ported everywhere. And the article itself doesn't even suggest using sqlite as a binary store. Rather, it suggests using the capabilities of sqlite to augment text-based data.
- sitkack 9y ago
  Using SQLite as a container is different from using the whole of SQLite the embedded sql database. You can use the container by reimplementing a reader for the file format. Maybe it would make the parent feel more comfortable if the sqlite db was inside a nice cozy zip (or jar file).
- mseebach 9y ago
  > sqlite files couldn't be more open Yes, actually they could. They could be readable without any software at all. Like XML. All of the improvements suggested in the article could be implemented by extending the XML format to allow including by other XML files by reference.
  - coldtea 9y ago
    >Yes, actually they could. They could be readable without any software at all. Like XML. You can't read XML without software. Perhaps you take editors and cat and less as a given. In which case, nothing prevents you from taking sqlite the binary (or 1000 serializers that can be produced very easily) as a given too.
  - chii 9y ago
    But you could also open the sqlite fine format using an sqlite client. The only difference is that xml is also human "readable". But most xml isn't really readable without a tool anyway.
    - mseebach 9y ago
      Yes, and that only difference is important, and it makes XML more open than SQLite. Perhaps not enough to matter (depending on your needs), but it is more open. You may not want to look at XML without a tool, but you could. Or, most likely, you could build a tool to look at the XML. This is a crucial design consideration of XML - even if all XML tooling, open or not, is lost forever, you can still read the XML. Not so with SQLite. Especially for a document format, this point seems important. That said, OpenDocument is zipped, which, while a simpler and even more ubiquitous format, is still a binary format that requires a library to read, so that detracts somewhat from the point. It's difficult to imagine a future where zip libraries survive but SQLite libraries don't.
      - jl6 9y ago
        Strictly speaking you can’t read XML without software either. Even a simple text editor is software of course, but you’ll also find XML files in arbitrary character encodings, and you absolutely do need non-trivial software to parse and display “plain” text in UTF-8.
        mseebach 9y ago
        You don't need a text editor, you just need to be able to read a sequence of bytes. XML anchors at the almost lowest possible point: you have the stream of characters. It's a pointless exercise to consider a format except on the premise that you have access to data in some way, and sequence of bytes is the simplest possible abstraction of data, a sequence of characters the second simplest. ASCII encoded text is simpler than UTF-8, but there is content that can't be represented in ASCII. Zip comes a bit up the food chain from there, and SQLite comes quite a bit further. XML is about being meaningful without tooling at the lowest possible level. Any added required tooling or documentation by definition makes the data less 'open'.
  - majewsky 9y ago
    > They could be readable without any software at all. They aren't, though. Or can you read ZIP archives without an unpacker?
    - geezerjay 9y ago
      Actually, iirc the OpenDocument format also includes an uncompressed, human readable, pure XML format.
      - mkesper 9y ago
        It's the 'flat' versions (fods etc.). Diff can be used on them.
      - _pmf_ 9y ago
        > Actually, iirc the OpenDocument format also includes an uncompressed, human readable, pure XML format. For some values of "readable" and "human".
    - masklinn 9y ago
      Writing a zip unpacker — including deflate decompressor — can be wrapped up in half a day by an average developer.
      - wruza 9y ago
        It sounds like good nuclear war proof solution. Maybe it is better to encode documents in Morse, because average radio man could transmit it via key in half a week.
        masklinn 9y ago
        > It sounds like good nuclear war proof solution. It fits the requirements of reading the file without (external) software since you can trivially reimplement it, and is also a much lower threshold than "you need to reimplement sqlite", which is why pretty much every system out there comes bundled with a zipfile unpacker.
  - jwr 9y ago
    This is a common recurring theme that I find strange. Why do people believe that if something can be browsed with less/Notepad, it is somehow more "readable" than a binary representation? I've seen and worked with tons of XML, and most of it was utterly useless without accompanying documentation, just as a binary format would be. The fact that I could "see" the XML on my screen did not make the job easier. The problem with formats "readable without any software" is that you usually end up with everything represented as strings, with no knowledge of actual types and constraints. So, you still need that documentation, even though the format is "readable without any software".
    - mseebach 9y ago
      Well, yes, this applies to reasonably well-designed, expressive XML (which OpenDocument is). You can definitely write XML that is perfectly incomprehensible without extensive references.
    - enord 9y ago
      It's not about human readability, it's about separating implementation from representation, so "readable without any software" is a good test for wether it's "readable with any sfotware". If your document editor represents documents as a stack of edit-events (for undo/redo or whatever) and you document every nook and cranny of this "document format", standardize it as "binary event-sourced document format, .esd" then your binary spec will be very close to a full description of your document editor. Everyone who wants to use your format now has to reimplement your editor. Anyone can leak implementation details into any format, and it is to some extent unavoidable, but if you can make it readable (or semi-readable), some thought has at least been put into the representation.
      - _pmf_ 9y ago
        > so "readable without any software" is a good test for wether it's "readable with any sfotware" What an extraordinarily stupid notion. Your filesystem is not readable without a filesystem driver for the specific binary format used to represent it, so you cannot even read your text file without having something in your stack that understands a more structured data organization scheme.
  - koonsolo 9y ago
    First of all, notepad is software. And second, xml is totally useless without software. But you do have a point that in a rare case, you could get your hands on the raw data which makes text easier. But everything is a tradeoff, and the question really is if the tradeoff in this case is worth it, since sqlite is so open and portable.
  - Chris2048 9y ago
    > extending the XML format XML is only trivially "readable without any software at all". The same is true of any binary format - the issue is that it can be read, but not easily processed - which is also the case for XML (you need to integrate an XML parser for that). And if you are including XML libs, why not SQlite libs?
    - fian 9y ago
      A printout of XML contains the data in a structured and understandable format. A printout of a binary dump does not provide any clues as to how the data was structured.
      - Chris2048 9y ago
        This is useful if you only need to read the data. If you need to process it in large amounts, its not so useful. And what if you need to traverse graph-like data in an XML format? > a binary dump does not provide any clues as to how the data was structured True, but if that binary format is SQlite, the structure is fully documented. You could work it out mentally just as you could work out any given XML schema.
    - geezerjay 9y ago
      The same is not true for any binary file. A hexdump of a plain text file is not as readable as opening the same plain text file with any basic text editor such as any notepad clone. That's the whole point: no one needs to install any software, let alone a specialized client, to read any human-readable format. More to the point, no one needs to install a full blown database engine to read a plain text document. You may argue that specialized tools make the job easier, but you also need to acknowledge that requiring specialized tools just to open a text file is silly.
  - PeachPlum 9y ago
    XML is UTF-8 you already need a decoder http://www.utf8-chartable.de/ http://www.utf8-chartable.de/
- josteink 9y ago
  > I don't see how your comment is responsive to the excellent article. doc and xls were proprietary binary formats. Indeed. He probably meant docx and xlsx.
  - mnw21cam 9y ago
    No, the point was that the doc and xls file formats were in themselves mini-filesystems, much like an SQLite file, and could support many of the same operations as described in the article. It however made them insanely complex to interpret. When the docx and xlsx formats were developed, they lost this capability.
- mcguire 9y ago
  The current Sqlite (post-3.0, 2004) file format: https://www.sqlite.org/fileformat.html https://www.sqlite.org/fileformat.html File Format Changes in SQLite: https://sqlite.org/formatchng.html https://sqlite.org/formatchng.html "Since 2004, there have been enhancements to SQLite such that newer database files are unreadable by older versions of the SQLite library. But the most recent versions of the SQLite library should be able to read and write any older SQLite database file without any problems." Me, I'm just happy all my old doccies are in TeX.
- da_chicken 9y ago
  Let's say you want to extract an image file from an .odt file and you haven't got Writer installed. You can do that by extracting the file from the .odt using unzip or similar. This article champions the idea that you should store the files as compressed binary BLOBs in SQLite. What does that mean you need to do? Connect to the file with SQLite, fetch the BLOB, hexdump it since it will return as a string, decompress it, and then save it to disk. Or save it to disk and then decompress it. You'd have to do something like this: https://stackoverflow.com/questions/15448373/how-to-dump-a-file-stored-in-a-sqlite-database-as-a-blob https://stackoverflow.com/questions/15448373/how-to-dump-a-f... That is what Open Document is supposed to end. This is not an excellent article. It's complaining that the Open Document Formats are bad because they don't use an optimized format. They're not supposed to be optimized. Documents aren't large enough to cause the types of problems that databases are intended to fix. They're supposed to be portable and easy to work with. They make it possible to access your data using the most basic computing tools: a ZIP archive tool and a text editor. That's the whole point.
  - asynchronous13 9y ago
    > It's complaining that the Open Document Formats are bad because they don't use an optimized format. That's the opposite of what the article states. From the introduction of the article: > Note that this is only a thought experiment. We are not suggesting that OpenDocument be changed. Nor is this article a criticism of the current OpenDocument design.
    - da_chicken 9y ago
      Yeah, and if you finish the quote, the next line is: > The point of this essay is to suggest ways to improve future file format designs. That makes it sound like, "No offense, but < extremely rude comment >." Just because they say they're not trying to be critical doesn't mean they're not actually being critical.
  - SQLite 9y ago
    If you have an image stored in an SQLite database file, you can extract it using the "sqlite3" command-line tool as described at https://www.sqlite.org/cli.html#file_i_o_functions https://www.sqlite.org/cli.html#file_i_o_functions Or, you could do it several dozen other ways. For example, maybe you have an archiver tool for SQLite that works like "zip" - https://www.sqlite.org/sqlar/doc/trunk/README.md https://www.sqlite.org/sqlar/doc/trunk/README.md About this much we agree: The objective is to make the content of the document more accessible and easier to use. The article intended to show that using an SQLite database rather than a pile-of-files in a ZIP archive achieves that goal. If my presentation were stored in an SQLite database, I might (depending on how the schema was structured of course) be able to ask interesting questions of the presentation, such as: * How many pages are in this presentation? * Which pages contain the word "discombobulated"? * How many images are in the presentation? * Does this presentation use a prepackaged background? * Does the presentation begin with my companies standard "start" slide? Can you write (say) a Python program that will answer any of the above about an OpenOffice .odp file? All I'm trying to say is that if OpenOffice .odp files were SQLite databases with a sensible schema, then probably you could answer any of the questions above with perhaps 5 lines line of code or less. Or just a shell script that uses the "sqlite3" command-line tool. As it stands now, if you want low-level information about an OpenOffice presentation, you have to open manually open the document on your disktop and click around - it cannot be (easily) automated. An ".odp" file is essentially an opaque BLOB that is not useful to me. Only the OpenOffice program (or its various forks) can access it. I have 104 historical presentations in a folder and I want to find some slide I wrote years ago. Right now, I have to laboriously open each presentation and manually search for the slide I want. If the format were SQLite, I could perhaps do the same with a query, which if run from a shell script, would quickly search all 104 presentations for me. So, Yes, the whole point is to make the content more easily accessible and usable. Relative to a pile-of-files in a ZIP archive, SQLite does exactly that.
- lloeki 9y ago
  doc and xls were even worse than proprietary binary formats: they were raw memory dumps of unserialized internal data structures. This meant implementing a doc opener was basically the tantalizing task of implementing Word-the-engine itself.

abtinf 9y ago

I don't see how your comment is responsive to the excellent article. doc and xls were proprietary binary formats. sqlite files couldn't be more open - the software to interact with them is in the public domain and ported everywhere. And the article itself doesn't even suggest using sqlite as a binary store. Rather, it suggests using the capabilities of sqlite to augment text-based data.

sitkack 9y ago
Using SQLite as a container is different from using the whole of SQLite the embedded sql database. You can use the container by reimplementing a reader for the file format. Maybe it would make the parent feel more comfortable if the sqlite db was inside a nice cozy zip (or jar file).
mseebach 9y ago
> sqlite files couldn't be more open Yes, actually they could. They could be readable without any software at all. Like XML. All of the improvements suggested in the article could be implemented by extending the XML format to allow including by other XML files by reference.
- coldtea 9y ago
  >Yes, actually they could. They could be readable without any software at all. Like XML. You can't read XML without software. Perhaps you take editors and cat and less as a given. In which case, nothing prevents you from taking sqlite the binary (or 1000 serializers that can be produced very easily) as a given too.
- chii 9y ago
  But you could also open the sqlite fine format using an sqlite client. The only difference is that xml is also human "readable". But most xml isn't really readable without a tool anyway.
  - mseebach 9y ago
    Yes, and that only difference is important, and it makes XML more open than SQLite. Perhaps not enough to matter (depending on your needs), but it is more open. You may not want to look at XML without a tool, but you could. Or, most likely, you could build a tool to look at the XML. This is a crucial design consideration of XML - even if all XML tooling, open or not, is lost forever, you can still read the XML. Not so with SQLite. Especially for a document format, this point seems important. That said, OpenDocument is zipped, which, while a simpler and even more ubiquitous format, is still a binary format that requires a library to read, so that detracts somewhat from the point. It's difficult to imagine a future where zip libraries survive but SQLite libraries don't.
    - jl6 9y ago
      Strictly speaking you can’t read XML without software either. Even a simple text editor is software of course, but you’ll also find XML files in arbitrary character encodings, and you absolutely do need non-trivial software to parse and display “plain” text in UTF-8.
      - mseebach 9y ago
        You don't need a text editor, you just need to be able to read a sequence of bytes. XML anchors at the almost lowest possible point: you have the stream of characters. It's a pointless exercise to consider a format except on the premise that you have access to data in some way, and sequence of bytes is the simplest possible abstraction of data, a sequence of characters the second simplest. ASCII encoded text is simpler than UTF-8, but there is content that can't be represented in ASCII. Zip comes a bit up the food chain from there, and SQLite comes quite a bit further. XML is about being meaningful without tooling at the lowest possible level. Any added required tooling or documentation by definition makes the data less 'open'.
- majewsky 9y ago
  > They could be readable without any software at all. They aren't, though. Or can you read ZIP archives without an unpacker?
  - geezerjay 9y ago
    Actually, iirc the OpenDocument format also includes an uncompressed, human readable, pure XML format.
    - mkesper 9y ago
      It's the 'flat' versions (fods etc.). Diff can be used on them.
    - _pmf_ 9y ago
      > Actually, iirc the OpenDocument format also includes an uncompressed, human readable, pure XML format. For some values of "readable" and "human".
  - masklinn 9y ago
    Writing a zip unpacker — including deflate decompressor — can be wrapped up in half a day by an average developer.
    - wruza 9y ago
      It sounds like good nuclear war proof solution. Maybe it is better to encode documents in Morse, because average radio man could transmit it via key in half a week.
      - masklinn 9y ago
        > It sounds like good nuclear war proof solution. It fits the requirements of reading the file without (external) software since you can trivially reimplement it, and is also a much lower threshold than "you need to reimplement sqlite", which is why pretty much every system out there comes bundled with a zipfile unpacker.
- jwr 9y ago
  This is a common recurring theme that I find strange. Why do people believe that if something can be browsed with less/Notepad, it is somehow more "readable" than a binary representation? I've seen and worked with tons of XML, and most of it was utterly useless without accompanying documentation, just as a binary format would be. The fact that I could "see" the XML on my screen did not make the job easier. The problem with formats "readable without any software" is that you usually end up with everything represented as strings, with no knowledge of actual types and constraints. So, you still need that documentation, even though the format is "readable without any software".
  - mseebach 9y ago
    Well, yes, this applies to reasonably well-designed, expressive XML (which OpenDocument is). You can definitely write XML that is perfectly incomprehensible without extensive references.
  - enord 9y ago
    It's not about human readability, it's about separating implementation from representation, so "readable without any software" is a good test for wether it's "readable with any sfotware". If your document editor represents documents as a stack of edit-events (for undo/redo or whatever) and you document every nook and cranny of this "document format", standardize it as "binary event-sourced document format, .esd" then your binary spec will be very close to a full description of your document editor. Everyone who wants to use your format now has to reimplement your editor. Anyone can leak implementation details into any format, and it is to some extent unavoidable, but if you can make it readable (or semi-readable), some thought has at least been put into the representation.
    - _pmf_ 9y ago
      > so "readable without any software" is a good test for wether it's "readable with any sfotware" What an extraordinarily stupid notion. Your filesystem is not readable without a filesystem driver for the specific binary format used to represent it, so you cannot even read your text file without having something in your stack that understands a more structured data organization scheme.
- koonsolo 9y ago
  First of all, notepad is software. And second, xml is totally useless without software. But you do have a point that in a rare case, you could get your hands on the raw data which makes text easier. But everything is a tradeoff, and the question really is if the tradeoff in this case is worth it, since sqlite is so open and portable.
- Chris2048 9y ago
  > extending the XML format XML is only trivially "readable without any software at all". The same is true of any binary format - the issue is that it can be read, but not easily processed - which is also the case for XML (you need to integrate an XML parser for that). And if you are including XML libs, why not SQlite libs?
  - fian 9y ago
    A printout of XML contains the data in a structured and understandable format. A printout of a binary dump does not provide any clues as to how the data was structured.
    - Chris2048 9y ago
      This is useful if you only need to read the data. If you need to process it in large amounts, its not so useful. And what if you need to traverse graph-like data in an XML format? > a binary dump does not provide any clues as to how the data was structured True, but if that binary format is SQlite, the structure is fully documented. You could work it out mentally just as you could work out any given XML schema.
  - geezerjay 9y ago
    The same is not true for any binary file. A hexdump of a plain text file is not as readable as opening the same plain text file with any basic text editor such as any notepad clone. That's the whole point: no one needs to install any software, let alone a specialized client, to read any human-readable format. More to the point, no one needs to install a full blown database engine to read a plain text document. You may argue that specialized tools make the job easier, but you also need to acknowledge that requiring specialized tools just to open a text file is silly.
- PeachPlum 9y ago
  XML is UTF-8 you already need a decoder http://www.utf8-chartable.de/ http://www.utf8-chartable.de/
josteink 9y ago
> I don't see how your comment is responsive to the excellent article. doc and xls were proprietary binary formats. Indeed. He probably meant docx and xlsx.
- mnw21cam 9y ago
  No, the point was that the doc and xls file formats were in themselves mini-filesystems, much like an SQLite file, and could support many of the same operations as described in the article. It however made them insanely complex to interpret. When the docx and xlsx formats were developed, they lost this capability.
mcguire 9y ago
The current Sqlite (post-3.0, 2004) file format: https://www.sqlite.org/fileformat.html https://www.sqlite.org/fileformat.html File Format Changes in SQLite: https://sqlite.org/formatchng.html https://sqlite.org/formatchng.html "Since 2004, there have been enhancements to SQLite such that newer database files are unreadable by older versions of the SQLite library. But the most recent versions of the SQLite library should be able to read and write any older SQLite database file without any problems." Me, I'm just happy all my old doccies are in TeX.
da_chicken 9y ago
Let's say you want to extract an image file from an .odt file and you haven't got Writer installed. You can do that by extracting the file from the .odt using unzip or similar. This article champions the idea that you should store the files as compressed binary BLOBs in SQLite. What does that mean you need to do? Connect to the file with SQLite, fetch the BLOB, hexdump it since it will return as a string, decompress it, and then save it to disk. Or save it to disk and then decompress it. You'd have to do something like this: https://stackoverflow.com/questions/15448373/how-to-dump-a-file-stored-in-a-sqlite-database-as-a-blob https://stackoverflow.com/questions/15448373/how-to-dump-a-f... That is what Open Document is supposed to end. This is not an excellent article. It's complaining that the Open Document Formats are bad because they don't use an optimized format. They're not supposed to be optimized. Documents aren't large enough to cause the types of problems that databases are intended to fix. They're supposed to be portable and easy to work with. They make it possible to access your data using the most basic computing tools: a ZIP archive tool and a text editor. That's the whole point.
- asynchronous13 9y ago
  > It's complaining that the Open Document Formats are bad because they don't use an optimized format. That's the opposite of what the article states. From the introduction of the article: > Note that this is only a thought experiment. We are not suggesting that OpenDocument be changed. Nor is this article a criticism of the current OpenDocument design.
  - da_chicken 9y ago
    Yeah, and if you finish the quote, the next line is: > The point of this essay is to suggest ways to improve future file format designs. That makes it sound like, "No offense, but < extremely rude comment >." Just because they say they're not trying to be critical doesn't mean they're not actually being critical.
- SQLite 9y ago
  If you have an image stored in an SQLite database file, you can extract it using the "sqlite3" command-line tool as described at https://www.sqlite.org/cli.html#file_i_o_functions https://www.sqlite.org/cli.html#file_i_o_functions Or, you could do it several dozen other ways. For example, maybe you have an archiver tool for SQLite that works like "zip" - https://www.sqlite.org/sqlar/doc/trunk/README.md https://www.sqlite.org/sqlar/doc/trunk/README.md About this much we agree: The objective is to make the content of the document more accessible and easier to use. The article intended to show that using an SQLite database rather than a pile-of-files in a ZIP archive achieves that goal. If my presentation were stored in an SQLite database, I might (depending on how the schema was structured of course) be able to ask interesting questions of the presentation, such as: * How many pages are in this presentation? * Which pages contain the word "discombobulated"? * How many images are in the presentation? * Does this presentation use a prepackaged background? * Does the presentation begin with my companies standard "start" slide? Can you write (say) a Python program that will answer any of the above about an OpenOffice .odp file? All I'm trying to say is that if OpenOffice .odp files were SQLite databases with a sensible schema, then probably you could answer any of the questions above with perhaps 5 lines line of code or less. Or just a shell script that uses the "sqlite3" command-line tool. As it stands now, if you want low-level information about an OpenOffice presentation, you have to open manually open the document on your disktop and click around - it cannot be (easily) automated. An ".odp" file is essentially an opaque BLOB that is not useful to me. Only the OpenOffice program (or its various forks) can access it. I have 104 historical presentations in a folder and I want to find some slide I wrote years ago. Right now, I have to laboriously open each presentation and manually search for the slide I want. If the format were SQLite, I could perhaps do the same with a query, which if run from a shell script, would quickly search all 104 presentations for me. So, Yes, the whole point is to make the content more easily accessible and usable. Relative to a pile-of-files in a ZIP archive, SQLite does exactly that.
lloeki 9y ago
doc and xls were even worse than proprietary binary formats: they were raw memory dumps of unserialized internal data structures. This meant implementing a doc opener was basically the tantalizing task of implementing Word-the-engine itself.