3 ms·

Another alternative that I've seen used is a zip or tar with no compression if you are just appending files and reading but only rarely updating or deleting. B

Another alternative that I've seen used is a zip or tar with no compression if you are just appending files and reading but only rarely updating or deleting. But sqlite is still better, it is more reliable, a bad write on that end of zip index destroys the whole zip archive and sqlite also gives you a lot more potential flexibility later on if you need to add metadata or something else. It is better in terms of inserts and deletes, although you will still need to vacuum.

hinkley 5y ago
A bad write on a zip file destroys the O(1) seek time, but it doesn't destroy the zip. That goes back to PKZip trying to work on floppies and over modems. You can still do an O(n) seek on a particular file, or expand and recompress the file to recover whatever isn't truncated. For this situation it does matter, but it is recoverable.
- asperous 5y ago
  Oh interesting! Good to know
  - hinkley 5y ago
    That property came back into use when we crossed the 2GB and then 4GB threshold for archives, because the TOC uses a 32 bit integer to reference backward into the file. A lot of implementations used a signed int, and even when they fixed it, organic growth in bandwidth and thus content size ran you into the wall in another couple of years. We had collectively mostly supported UTF-8 filenames around that same time period, so it was going from one problem to the next (or choosing between implementations that had fixed one but not the other). For the 32 bit address problem, you could read forward from the front, and as long as no entry was over 4GB, you could still read the file. If the file count was low enough you could cache all of the entry objects, and if reads dominated opens, then you were functionally back to O(1) access (but O(n) startup). There was a 64 bit extension going around, but when I stopped working with zip files every day I stopped tracking the progress.

hinkley 5y ago

A bad write on a zip file destroys the O(1) seek time, but it doesn't destroy the zip. That goes back to PKZip trying to work on floppies and over modems. You can still do an O(n) seek on a particular file, or expand and recompress the file to recover whatever isn't truncated. For this situation it does matter, but it is recoverable.

asperous 5y ago
Oh interesting! Good to know
- hinkley 5y ago
  That property came back into use when we crossed the 2GB and then 4GB threshold for archives, because the TOC uses a 32 bit integer to reference backward into the file. A lot of implementations used a signed int, and even when they fixed it, organic growth in bandwidth and thus content size ran you into the wall in another couple of years. We had collectively mostly supported UTF-8 filenames around that same time period, so it was going from one problem to the next (or choosing between implementations that had fixed one but not the other). For the 32 bit address problem, you could read forward from the front, and as long as no entry was over 4GB, you could still read the file. If the file count was low enough you could cache all of the entry objects, and if reads dominated opens, then you were functionally back to O(1) access (but O(n) startup). There was a 64 bit extension going around, but when I stopped working with zip files every day I stopped tracking the progress.