4 ms·

Based on personal experience on enterprise storage solutions, LZO is the best generic compressor. Yeah, LZO.

rincebrain 4y ago
By which metrics? Usually, when I see people claim things that are mutually inconsistent about compression, it turns out they have very different tradeoffs they want to do for decompression speed {avg,worst}/compression speed {avg,worst}/compression ratio.
- OnlyMortal 4y ago
  Compression ratio wrt CPU usage (time).
prtkgpt 4y ago
londons_explore 4y ago
This is a good chart to have in mind: https://community.centminmod.com/data/attachment-files/2019/04/7890_round3-part4-charted-shorter-02.png https://community.centminmod.com/data/attachment-files/2019/...
- klauspost 4y ago
  Here is an updated chart - and the data (2019): https://community.centminmod.com/threads/round-4-compression-comparison-benchmarks-zstd-vs-brotli-vs-pigz-vs-bzip2-vs-xz-etc.18669/ https://community.centminmod.com/threads/round-4-compression... Yes, though decompression speed (+memory to some extent) should also be considered. Also different input types has different characteristics, so a single type isn't very representative. While ultimate compression efficiency is interesting, in practice I find that most often you are looking for a "good enough", meaning "give me the best at xxxMB/s". For most purposes (de)compression at 50MB/s is just too slow to bother. For a server anything considerably below 1GB/s/core is too slow, otherwise you will too easily saturate your CPU before your NIC or disks are saturated.
  - londons_explore 4y ago
    Well according to the chart, "give me the best at 1Gb/sec" would seem to be zstd level 2... Also, zstd seems to operate over the widest range - a simple algorithm to use that would be: "Each second, check... Is network utilization over 90% and CPU utilization under 90%? If so, increase compression level by one. If the reverse, decrease compression level by one"

By which metrics? Usually, when I see people claim things that are mutually inconsistent about compression, it turns out they have very different tradeoffs they want to do for decompression speed {avg,worst}/compression speed {avg,worst}/compression ratio.

OnlyMortal 4y ago
Compression ratio wrt CPU usage (time).

prtkgpt 4y ago

londons_explore 4y ago

This is a good chart to have in mind: https://community.centminmod.com/data/attachment-files/2019/04/7890_round3-part4-charted-shorter-02.png https://community.centminmod.com/data/attachment-files/2019/...

klauspost 4y ago
Here is an updated chart - and the data (2019): https://community.centminmod.com/threads/round-4-compression-comparison-benchmarks-zstd-vs-brotli-vs-pigz-vs-bzip2-vs-xz-etc.18669/ https://community.centminmod.com/threads/round-4-compression... Yes, though decompression speed (+memory to some extent) should also be considered. Also different input types has different characteristics, so a single type isn't very representative. While ultimate compression efficiency is interesting, in practice I find that most often you are looking for a "good enough", meaning "give me the best at xxxMB/s". For most purposes (de)compression at 50MB/s is just too slow to bother. For a server anything considerably below 1GB/s/core is too slow, otherwise you will too easily saturate your CPU before your NIC or disks are saturated.
- londons_explore 4y ago
  Well according to the chart, "give me the best at 1Gb/sec" would seem to be zstd level 2... Also, zstd seems to operate over the widest range - a simple algorithm to use that would be: "Each second, check... Is network utilization over 90% and CPU utilization under 90%? If so, increase compression level by one. If the reverse, decrease compression level by one"