4 ms·

Very cool idea! I'm a bit torn whether the format should concern itself with compression. Seems like a useful general blob container strategy, might be prudent

Very cool idea! I'm a bit torn whether the format should concern itself with compression. Seems like a useful general blob container strategy, might be prudent to leave compression to the consumer?

genocidicbunny 5y ago
I've hand-rolled something very similar to this, and I just used an extra 'tag' column that stored some extra info, often the compression type. FWIW, at the bottom of the linked page it says that you can skip the sqlar_compress/uncompress functions and just roll your own: > The code above is for the general case. For the special case of an SQLite Archive that only stores uncompressed or uncompressible content (this might come up, for example, in an SQLite Archive that stores only JPEG, GIF, and/or PNG images) then the content can be inserted into and extracted from the database without using the sqlar_compress() and sqlar_uncompress() functions, and the sqlar.c extension is not required.
- kybernetikos 5y ago
  That seems like a nice effect of storing the uncompressed size along with the blob, and using that to control decompression. You can always choose to store uncompressed blobs, and the size will always be the same so sqlite will not attempt to decompress the blob. It's not quite the same as rolling your own compression, but it's pretty easy to see how the format could be extended to support that.
  - genocidicbunny 5y ago
    Using the size to trigger compression vs no-op is clever, but doesn't help when you have different compression methods that you want to use. For example, you be using this as an archive format for a game, and you may want to store the texture data using a different compression method than the ai scripts or the audio data. At that point, you definitely need another piece of data to clue you in. A tag field is useful then, though I've also seen filename/file extension-based methods as well.

Very cool idea! I'm a bit torn whether the format should concern itself with compression. Seems like a useful general blob container strategy, might be prudent to leave compression to the consumer?

genocidicbunny 5y ago

I've hand-rolled something very similar to this, and I just used an extra 'tag' column that stored some extra info, often the compression type. FWIW, at the bottom of the linked page it says that you can skip the sqlar_compress/uncompress functions and just roll your own: > The code above is for the general case. For the special case of an SQLite Archive that only stores uncompressed or uncompressible content (this might come up, for example, in an SQLite Archive that stores only JPEG, GIF, and/or PNG images) then the content can be inserted into and extracted from the database without using the sqlar_compress() and sqlar_uncompress() functions, and the sqlar.c extension is not required.

kybernetikos 5y ago
That seems like a nice effect of storing the uncompressed size along with the blob, and using that to control decompression. You can always choose to store uncompressed blobs, and the size will always be the same so sqlite will not attempt to decompress the blob. It's not quite the same as rolling your own compression, but it's pretty easy to see how the format could be extended to support that.
- genocidicbunny 5y ago
  Using the size to trigger compression vs no-op is clever, but doesn't help when you have different compression methods that you want to use. For example, you be using this as an archive format for a game, and you may want to store the texture data using a different compression method than the ai scripts or the audio data. At that point, you definitely need another piece of data to clue you in. A tag field is useful then, though I've also seen filename/file extension-based methods as well.