4 ms·

This is pretty cool. If it has low enough overhead, I wonder if this could be used to replace the current format of ODT (Open Document) files, which right now a

by warent 9y ago

This is pretty cool. If it has low enough overhead, I wonder if this could be used to replace the current format of ODT (Open Document) files, which right now are basically just zipped up XML documents and image assets

WorldMaker 9y ago
Looking at the file format limitations [1], it doesn't seem like a good fit for an alternative to ODT files. (A cursory judgment based solely on this wiki page has me wondering if they are using the best data structure for an Index, but again, very cursory.) A question is why you would want to replace the current ODT format? "Just" a zipped up file tree of XML documents and image assets is a pretty robust format choice. There are unzip programs on almost every platform. XML isn't always everyone's favorite markup language, but it's well supported by almost every language ecosystem, and still roughly okay to humanly eyeball if needed. The relative inefficiencies of the textual encoding and verbosity of XML are directly suited for compression efficiency of a Zip file. For most trade-offs of a document file, it's a somewhat resilient option, with likely a long shelf life (both zip and XML technologies have lasted enough decades, and are reasonably well standardized enough, that you can imagine they will continue to be unzippable/readable decades from now). [1] https://github.com/mbdavid/LiteDB/wiki/How-LiteDB-Works https://github.com/mbdavid/LiteDB/wiki/How-LiteDB-Works
- pletnes 9y ago
  Well suited? Zip? I’ve tested some hacks to get a file below email server file size limitations. Unzipping an excel file, then tarballing it into a .tar.xz archive can cut the size by 30%, say. The only issue is that the receiver might not know what on earth to do with it!
  - WorldMaker 9y ago
    "Well suited" doesn't necessarily mean "best suited". :)
  - discreditable 9y ago
    Another trick is you can repack modern MS Office files with advzip from advancecomp. With -z4 It will use Zopfli to pack them. Further gains can be had with more rounds, (such as -i50). The resulting files can be quite a bit smaller and Office will open them without complaint. Yet another trick is you can save as binary format files. In Excel this is XLSB. Supposedly those are less compatible across Office versions however.
- gwbas1c 9y ago
  Because incremental updates are very difficult in a zip file. The point of an embedded database like LiteDB or SQLite is to allow incremental updates to a file without needing to write out the entire file every time. The nice thing about tools like SQLite and LiteDB is that they have APIs that are fundamentally similar to APIs that we already know. It keeps the learning curve light so we can spend more time on the task at hand instead of learning a new thing.
  - sametmax 9y ago
    Give the size of most ODT files and the power of current computers, incremental updates are completely irrelevant. Regenerating the entire file takes a blink. But the current format benefit from the XML ecosystem (validation, transformation, namespacing, nesting, semantic markup) that is very handy for a document. Plus, being text based is a very good selling point if you want your format to last: event in 30 years if you don't have the spec, you can peak in the zip and see what's up.
lostapathy 9y ago
sqlite has already done a similar thought experiment: https://sqlite.org/affcase1.html https://sqlite.org/affcase1.html

by warent 9y ago

WorldMaker 9y ago

Looking at the file format limitations [1], it doesn't seem like a good fit for an alternative to ODT files. (A cursory judgment based solely on this wiki page has me wondering if they are using the best data structure for an Index, but again, very cursory.) A question is why you would want to replace the current ODT format? "Just" a zipped up file tree of XML documents and image assets is a pretty robust format choice. There are unzip programs on almost every platform. XML isn't always everyone's favorite markup language, but it's well supported by almost every language ecosystem, and still roughly okay to humanly eyeball if needed. The relative inefficiencies of the textual encoding and verbosity of XML are directly suited for compression efficiency of a Zip file. For most trade-offs of a document file, it's a somewhat resilient option, with likely a long shelf life (both zip and XML technologies have lasted enough decades, and are reasonably well standardized enough, that you can imagine they will continue to be unzippable/readable decades from now). [1] https://github.com/mbdavid/LiteDB/wiki/How-LiteDB-Works https://github.com/mbdavid/LiteDB/wiki/How-LiteDB-Works

pletnes 9y ago
Well suited? Zip? I’ve tested some hacks to get a file below email server file size limitations. Unzipping an excel file, then tarballing it into a .tar.xz archive can cut the size by 30%, say. The only issue is that the receiver might not know what on earth to do with it!
- WorldMaker 9y ago
  "Well suited" doesn't necessarily mean "best suited". :)
- discreditable 9y ago
  Another trick is you can repack modern MS Office files with advzip from advancecomp. With -z4 It will use Zopfli to pack them. Further gains can be had with more rounds, (such as -i50). The resulting files can be quite a bit smaller and Office will open them without complaint. Yet another trick is you can save as binary format files. In Excel this is XLSB. Supposedly those are less compatible across Office versions however.
gwbas1c 9y ago
Because incremental updates are very difficult in a zip file. The point of an embedded database like LiteDB or SQLite is to allow incremental updates to a file without needing to write out the entire file every time. The nice thing about tools like SQLite and LiteDB is that they have APIs that are fundamentally similar to APIs that we already know. It keeps the learning curve light so we can spend more time on the task at hand instead of learning a new thing.
- sametmax 9y ago
  Give the size of most ODT files and the power of current computers, incremental updates are completely irrelevant. Regenerating the entire file takes a blink. But the current format benefit from the XML ecosystem (validation, transformation, namespacing, nesting, semantic markup) that is very handy for a document. Plus, being text based is a very good selling point if you want your format to last: event in 30 years if you don't have the spec, you can peak in the zip and see what's up.

lostapathy 9y ago

sqlite has already done a similar thought experiment: https://sqlite.org/affcase1.html https://sqlite.org/affcase1.html