3 ms·
I hope they pay greater attention to the low and high end of their compression ratio spectrum. On the low end, it'd be great if it could exceed lz4 in terms of
by m0zg 8y ago
I hope they pay greater attention to the low and high end of their compression ratio spectrum. On the low end, it'd be great if it could exceed lz4 in terms of speed and memory savings. On the high end it'd be great to exceed XZ/LZMA.
Right now it's impressive "in the middle", but I find myself in a lot of situations where I care about the extremes. I.e. for something that will be transferred a lot, or cold-stored, I want maximum compression, CPU/RAM usage be damned, within reason. So I tend to use LZMA there if files aren't too large. For realtime/network RPC scenarios I want minimum RAM/CPU usage and Pareto-optimality on multi-GbE networks. This is where I use LZ4 (and used to use Snappy/Zippy).
At their scale, though, FB is surely saving many millions of dollars thanks to deploying this, both in human/machine time savings and storage savings.
- terrelln 8y agoWe recently added negative compression levels that extends the fast end of the spectrum significantly. We are also working on incrementally improving the strong end of the compression ratio spectrum. We don't expect plain zstd to compress stronger than xz, but we hope to close the gap some.
- m0zg 8y agoAh, I didn't know, thanks! I'll have to re-test.
- JyrkiAlakuijala 8y agoWith zstd you end up on average 5-6 % worse than LZMA in density on a variety of large test corpora, but decodes 8x faster. Brotli with --large_window 30 can get within 0.6 % of LZMA, and decodes 5x faster than LZMA. https://encode.ru/threads/2947-large-window-brotli-results-available-for-a-several-corpora https://encode.ru/threads/2947-large-window-brotli-results-a...
- praseodym 8y agoIt does depend on your dataset. In my experiments, Zstandard did outperform XZ/LZMA in terms of compression ratio, but not by much. However, the massive improvement in decompression speed is what won me over.