3 ms·
Zstd (https://facebook.github.io/zstd/ https://facebook.github.io/zstd/) provides more flexibility and supports user provided compression dictionaries - you can
by d110af5ccf 5y ago
Zstd (https://facebook.github.io/zstd/ https://facebook.github.io/zstd/) provides more flexibility and supports user provided compression dictionaries - you can just place one at the front of the archive.
Of course this doesn't fully solve the metadata issue and it's definitely more complex than just using the zip format. But on the plus side, it allows you to avoid using the zip format and the compression algorithm is significantly more performant and flexible.
- stouset 5y agoPlacing a user-supplied dictionary at the front of the archive is essentially how “normal” compression works. The advantage of a user-supplied dictionary is that it can be transmitted once, out of band. With this approach it can also be larger than what would make sense for an individual file and it can take advantage of similarity between files. But if you just send it along with the compressed data anyway, you’re not buying anything extra.
- d110af5ccf 5y agoYes, I realize that it's manually reinventing the wheel. However the context here was supporting random access (and append, truncate, etc) while also using separate formats for archiving and compression. Such an approach is more work but provides clean separation and thus far more flexibility. Bear in mind that the alternative (in context) is throwing a bunch of independently compressed files at something like tar. In the event that the files are small, that will absolutely destroy the compression ratio. If the files are large enough or you only have a few then you could always omit the custom dictionary.
- mort96 5y agoTo manage this, you're essentially back to writing a tool which handles both compression and zipping, which is exactly what zzo38computer set out to avoid.