3 ms·

RAD's packer is nice for cases when compression time is not the most important aspect. Seems like LZ4 is still unbeatable if you need as fast compression as po

by vardump 10y ago

RAD's packer is nice for cases when compression time is not the most important aspect. Seems like LZ4 is still unbeatable if you need as fast compression as possible. Think kernel memory page compression (to avoid swapping pages to disk) and compressing data destined to a high bandwidth sink. I'm just guessing, but probably LZ4 has also less power consumption. If your output device can do over ~100 megabytes per second, LZ4 seems to win easily. Optimal compression algorithm selection seems to depend on capacity and bandwidth constraints of output device.

aktau 10y ago
I often use a script I hacked together called memo [1] when exploring the commandline. Today I was running a pipeline like this: $ memo expensive-with-lots-of-output | memo -s expensive | less I noticed that lz4 (the default (de)compressor memo uses) was the bottleneck. ("memo -s" absorbs standard input and calculates a checksum over it and the command arguments, if that checksum is in the cache, it knows it can just output the cached value instead of running the command). So I started looking around if there was nothing better around. From the Squish benchmarks posted elsewhere on this article [2], I noticed that zstd(1) seemed to make the right trade-offs. I required compression speed > 50MB/s, decompression speed at least as fast, and as high a compression ration as possible. zstd(1) seems to work brilliantly and is very fast, it was easy to compile too, just running make in its directory was sufficient for me. I love those kinds of projects. I've made it the default over lz4. What gave me even more confidence in this change is that zstd is by the author of lz4. [1]: https://github.com/aktau/dotfiles/blob/master/bin/memo https://github.com/aktau/dotfiles/blob/master/bin/memo (currently a version that doesn't prefer zstd(1) yet, I need to git push). [2]: https://quixdb.github.io/squash-benchmark/unstable/ https://quixdb.github.io/squash-benchmark/unstable/
- vardump 10y ago
  > I noticed that lz4 (the default (de)compressor memo uses) was the bottleneck. It probably depend on how much bandwidth you got.
fulafel 10y ago
The RAD guys have entries in that dept too. "Selkie is 1.3X faster to decode than LZ4 and gets 12% more compression" [1] though "Selkie does not currently have a super fast encoder" [2] [1] http://cbloomrants.blogspot.com/2016/07/introducing-oodle-mermaid-and-selkie.html http://cbloomrants.blogspot.com/2016/07/introducing-oodle-me... [2] http://cbloomrants.blogspot.nl/2016/07/oodle-selkie.html http://cbloomrants.blogspot.nl/2016/07/oodle-selkie.html

by vardump 10y ago

aktau 10y ago

I often use a script I hacked together called memo [1] when exploring the commandline. Today I was running a pipeline like this: $ memo expensive-with-lots-of-output | memo -s expensive | less I noticed that lz4 (the default (de)compressor memo uses) was the bottleneck. ("memo -s" absorbs standard input and calculates a checksum over it and the command arguments, if that checksum is in the cache, it knows it can just output the cached value instead of running the command). So I started looking around if there was nothing better around. From the Squish benchmarks posted elsewhere on this article [2], I noticed that zstd(1) seemed to make the right trade-offs. I required compression speed > 50MB/s, decompression speed at least as fast, and as high a compression ration as possible. zstd(1) seems to work brilliantly and is very fast, it was easy to compile too, just running make in its directory was sufficient for me. I love those kinds of projects. I've made it the default over lz4. What gave me even more confidence in this change is that zstd is by the author of lz4. [1]: https://github.com/aktau/dotfiles/blob/master/bin/memo https://github.com/aktau/dotfiles/blob/master/bin/memo (currently a version that doesn't prefer zstd(1) yet, I need to git push). [2]: https://quixdb.github.io/squash-benchmark/unstable/ https://quixdb.github.io/squash-benchmark/unstable/

vardump 10y ago
> I noticed that lz4 (the default (de)compressor memo uses) was the bottleneck. It probably depend on how much bandwidth you got.

fulafel 10y ago

The RAD guys have entries in that dept too. "Selkie is 1.3X faster to decode than LZ4 and gets 12% more compression" [1] though "Selkie does not currently have a super fast encoder" [2] [1] http://cbloomrants.blogspot.com/2016/07/introducing-oodle-mermaid-and-selkie.html http://cbloomrants.blogspot.com/2016/07/introducing-oodle-me... [2] http://cbloomrants.blogspot.nl/2016/07/oodle-selkie.html http://cbloomrants.blogspot.nl/2016/07/oodle-selkie.html