3 ms·

> Specifically sqlar is a worse replacement of ZIP. I mostly agree. However, it does have some advantages too, as mentioned in the linked article. (An alterna

> Specifically sqlar is a worse replacement of ZIP. I mostly agree. However, it does have some advantages too, as mentioned in the linked article. (An alternative way to query archives by SQL would be to implement a virtual table; in this case it can work with any archive format. In the case of ZIP format, it already does this.) > filesystem and custom metadata besides from simple st_mode Since it is a SQLite database, you can add additional tables etc, so it isn't a problem. > solid compression, metadata compression I think that using separate archiving and compression is good, which would do these things. A separate program should be used for archive and for compression. However, in some circumstances, solid compression is undesirable (as mentioned before in HN). In some formats, solid compression is optional, but you might want an index, and there will need to be some way to find parts of the file by index. This is why in my own experimental concatenatable compression format, you can add an optional key frame index. > encryption Also encryption should be a separate program from archive and compression too, I should think. (However, there are extensions for SQLite for doing encryption, so it is possible to use those if wanted.) > the compression mode (DEFLATE vs. uncompressed) is implicitly indicated by `sz = length(data)` and I don't think it is a good idea I agree; it doesn't look like good to me either. (Although, the Free Hero Mesh code page file does something similar, but in that case the uncompressed sizes of lumps is always exactly 2048 bytes anyways, so there is no need to store a separate uncompressed size.)

lifthrasiir 5y ago
> Since it is a SQLite database, you can add additional tables etc, so it isn't a problem. You can add additional tables, but you can't add support for additional tables in the past versions. This is especially true because sqlar has no version specified. > I think that using separate archiving and compression is good, which would do these things. A separate program should be used for archive and for compression. I've heard this argument a lot and I think it is completely false. Archival and compression should go together, otherwise you would get suboptimal results. Archival then compression (i.e. the traditional tarballs) will disable random access to individual files and metadata. This hurts performance in pretty every use case except for the one-shot decompression. In particular this problem completely prohibits using archival then compression in container formats. You can reenable them by using random-accessible compression formats, but they are not common and less efficient than formats without that constraint. Compression then archival is much better than archival then compression because it essentially mimics the combined compression-archival format (I have used .bz2.tar in the past), but this mode is unpopular so you need to manually decompress them. And it is harder to use different compression algorithms for different formats; you should put that info to the file name or the metadata, and that would be no different from compression-archival formats. Some claim that the needs for solid compression demonstrate the problem of combined compression-archival formats. This is not true and even misleading. Archival then compression commonly results in worse compression than solid compressed archives even with the same algorithm. The gist of solid compression is a shared context for related files, so unless the archiver somehow reorders files it can't compete with combined archivers with solid compression. And if your archiver is doing that reordering, it is no longer decoupled from the compressor so it would be better to combine them. In addition to this, there are also alternatives to solid compression with different trade-offs, for example learned shared preset dictionaries (even zlib supports this, although unpopular). > Also encryption should be a separate program from archive and compression too, I should think. Yeah, I'm personally fine without encryption as it is really hard to get it right (for example, you should be prepared of metadata encryption and key stretching). I've listed it partly for the completeness and partly for the fact that it is probably the most approachable encryption software to end users. > (However, there are extensions for SQLite for doing encryption, so it is possible to use those if wanted.) And I think the official encryption extension [1] is proprietary. [1] https://www.sqlite.org/see/ https://www.sqlite.org/see/

lifthrasiir 5y ago

> Since it is a SQLite database, you can add additional tables etc, so it isn't a problem. You can add additional tables, but you can't add support for additional tables in the past versions. This is especially true because sqlar has no version specified. > I think that using separate archiving and compression is good, which would do these things. A separate program should be used for archive and for compression. I've heard this argument a lot and I think it is completely false. Archival and compression should go together, otherwise you would get suboptimal results. Archival then compression (i.e. the traditional tarballs) will disable random access to individual files and metadata. This hurts performance in pretty every use case except for the one-shot decompression. In particular this problem completely prohibits using archival then compression in container formats. You can reenable them by using random-accessible compression formats, but they are not common and less efficient than formats without that constraint. Compression then archival is much better than archival then compression because it essentially mimics the combined compression-archival format (I have used .bz2.tar in the past), but this mode is unpopular so you need to manually decompress them. And it is harder to use different compression algorithms for different formats; you should put that info to the file name or the metadata, and that would be no different from compression-archival formats. Some claim that the needs for solid compression demonstrate the problem of combined compression-archival formats. This is not true and even misleading. Archival then compression commonly results in worse compression than solid compressed archives even with the same algorithm. The gist of solid compression is a shared context for related files, so unless the archiver somehow reorders files it can't compete with combined archivers with solid compression. And if your archiver is doing that reordering, it is no longer decoupled from the compressor so it would be better to combine them. In addition to this, there are also alternatives to solid compression with different trade-offs, for example learned shared preset dictionaries (even zlib supports this, although unpopular). > Also encryption should be a separate program from archive and compression too, I should think. Yeah, I'm personally fine without encryption as it is really hard to get it right (for example, you should be prepared of metadata encryption and key stretching). I've listed it partly for the completeness and partly for the fact that it is probably the most approachable encryption software to end users. > (However, there are extensions for SQLite for doing encryption, so it is possible to use those if wanted.) And I think the official encryption extension [1] is proprietary. [1] https://www.sqlite.org/see/ https://www.sqlite.org/see/