3 ms·
https://github.com/facebook/zstd/issues/395#issuecomment-535875379 https://github.com/facebook/zstd/issues/395#issuecomment-535... If anyone from Facebook is r
by rwmj 6y ago
https://github.com/facebook/zstd/issues/395#issuecomment-535875379 https://github.com/facebook/zstd/issues/395#issuecomment-535...
If anyone from Facebook is reading, please look at making zstd seekable. There is already experimental support but it needs to be standardized. This could open up really interesting new use-cases [see my comment linked above], where currently we have to use much slower xz/lzma instead.
- saurik 6y agoSo, xz isn't as much of a compression algorithm as a compression container format (which is a big reason why it generates slightly larger files than directly using the lzma tooling, and also the reason why a lot of people dislike it for solving a kind of weird problem); why not just add a filter type for zstd to xz? Adding this support to zstd when we already have this awkward container format to solve this specific problem in isolation seems like the wrong abstraction.
- rwmj 6y agoSure - the problem isn't really on a technical level, it's the social problem of getting everyone to agree to adhere to one standard and then we can implement that in a single set of formats and tools.
- seppel 6y agoYou can use sqlite with zstd compression (which can be extremely efficient with a shared compression dictionary).
- phiresky 6y agoI'm actually working on a sqlite addon right now that allows to transparently compress tables in sqlite using zstd dictionaries and the preliminary results are really promising! One DB I have compresses down from 1GB to 80MB and random SELECTs are faster in the compressed DB when the cache is cold and only somewhat slower when it is hot :) https://github.com/phiresky/sqlite-zstd https://github.com/phiresky/sqlite-zstd