5 ms·

Would be interesting if this was expanded to support more compression formats. E.g. zstd. Gzip is quite an old format and zstd is a lot quicker to decompress.

chrisseaton 5y ago
I don't think it uses the gzip format does it? It just uses deflate.
- lifthrasiir 5y ago
  It's actually the zlib format. The linked post says DEFLATE, but the linked source code correctly mentions zlib instead.
- masklinn 5y ago
  It doesn't use the gzip file format, but it uses the gzip compression format (which is just DEFLATE).
rini17 5y ago
Yes I did double check there's no column to indicate compressed format, no idea how they want to extend it in future. Maybe just autodetect from the contents? However that could fail if you want to store already deflated files. I'd also like (nullable) mimetype column. Can be handy for example to store encoding for text files. On the other side, come people call for file permissions... I'm not a fan, too system-dependent.
- mark-r 5y ago
  Since compression can produce any bit combination with equal probability, the only way to reliably auto-detect from the contents is to put some kind of fixed flag at the front. At that point you get more flexibility by making it a new column in the table instead. It doesn't seem like it would be hard to extend this with a compression type and mime type if you need those. I question the need for multiple compression types though, the difference between the default and the best won't be great enough to be worth the hassle. Especially if it leaves you open to a patent troll.
  - edwintorok 5y ago
    `zstd` and `zlib` as the only 2 supported formats would be enough for me (zstd for state of the art, fast compression and decompression, with choice of high compression ratios if needed; and zlib for backward compatibility/portability), but others may disagree and want their own favourite compression format there. OTOH if you use zstd that already has multiple file capability built-in, so you don't gain much by using sqlite (unless zstd were to adopt sqlite as.its storage format...).
    - edwintorok 5y ago
      Actually looks like if you pass multiple files to zstd it'll compress them individually, it won't create an archive, so tar or sqlite format would still be useful there.

chrisseaton 5y ago

I don't think it uses the gzip format does it? It just uses deflate.

lifthrasiir 5y ago
It's actually the zlib format. The linked post says DEFLATE, but the linked source code correctly mentions zlib instead.
masklinn 5y ago
It doesn't use the gzip file format, but it uses the gzip compression format (which is just DEFLATE).

rini17 5y ago

Yes I did double check there's no column to indicate compressed format, no idea how they want to extend it in future. Maybe just autodetect from the contents? However that could fail if you want to store already deflated files. I'd also like (nullable) mimetype column. Can be handy for example to store encoding for text files. On the other side, come people call for file permissions... I'm not a fan, too system-dependent.

mark-r 5y ago
Since compression can produce any bit combination with equal probability, the only way to reliably auto-detect from the contents is to put some kind of fixed flag at the front. At that point you get more flexibility by making it a new column in the table instead. It doesn't seem like it would be hard to extend this with a compression type and mime type if you need those. I question the need for multiple compression types though, the difference between the default and the best won't be great enough to be worth the hassle. Especially if it leaves you open to a patent troll.
- edwintorok 5y ago
  `zstd` and `zlib` as the only 2 supported formats would be enough for me (zstd for state of the art, fast compression and decompression, with choice of high compression ratios if needed; and zlib for backward compatibility/portability), but others may disagree and want their own favourite compression format there. OTOH if you use zstd that already has multiple file capability built-in, so you don't gain much by using sqlite (unless zstd were to adopt sqlite as.its storage format...).
  - edwintorok 5y ago
    Actually looks like if you pass multiple files to zstd it'll compress them individually, it won't create an archive, so tar or sqlite format would still be useful there.