7 ms·

Lzbench compression benchmark

23 points by todsacerdoti 2y ago

pella 2y ago
Probably, the results in the table are about 7 years old. For example, the table shows version 1.3.4 for zstd, which indicates that the results are outdated - currently, version 1.5.6 is the latest release ( https://github.com/facebook/zstd/releases/tag/v1.5.6 https://github.com/facebook/zstd/releases/tag/v1.5.6 ) In my opinion, it is better to check the original repository: https://github.com/inikep/lzbench https://github.com/inikep/lzbench or this: https://github.com/inikep/lzbench/blob/master/doc/lzbench20_sorted.md https://github.com/inikep/lzbench/blob/master/doc/lzbench20_...
- orthoxerox 2y ago
  Judging by the last link, the only options that are strictly superior to snappy in every sense are lz4 (as OP claimed) and zstd (levels -3 to +1). If compression speed is critical, then the default level of zstd compression in most big data tools (+3) might be the wrong one for you, but test on your own data, not Silesia.
gotoeleven 2y ago
Based on these results, why would you use anything but lz4_fast ? Are there other dimensions that might be important for a particular use case so a different algo may be best? Ones I can think of might be multi threaded or SIMD friendliness, cache efficiency, memory efficiency, information locality, stream-ability
- pella 2y ago
  latest - sorted by Ratio : https://github.com/inikep/lzbench/blob/master/doc/lzbench20_sorted.md https://github.com/inikep/lzbench/blob/master/doc/lzbench20_...
- Galanwe 2y ago
  > Are there other dimensions that might be important for a particular use case so a different algo may be best? Choosing a compression/decompression algo is an optimization problem essentially. It mainly depends on your IO for input. For instance, if you have very fast IO, say a local SSD at 7GB/s, then you probably don't want compression, as even the fastest decompression will not go beyond 4GB/s. If you have moderately fast IO, say 4GB/s, then lz4 is likely a good candidate, you will get maybe a 0.5 compression ratio, meaning you will spend half the time reading your input, while decompressing at nearly the same same speed that you would read uncompressed data. If you have very slow IO, say 100MB/s, then the compression ratio becomes the main factor, overriding decompression speed. So, essentially, you want to solve an optimization problem of lowering the time to read the compressed file and decompressing it, conditioned by your reading speed, writing speed, and CPU to spare, under a time constraint of compression speed. There are solvers on internet doe this problem, but I would love to see someone write it down here in plain math.
nolist_policy 2y ago
The data sets are way too small, the biggest one (silesia/mozilla) is just 50mb.

23 points by todsacerdoti 2y ago

Probably, the results in the table are about 7 years old. For example, the table shows version 1.3.4 for zstd, which indicates that the results are outdated - currently, version 1.5.6 is the latest release ( https://github.com/facebook/zstd/releases/tag/v1.5.6 https://github.com/facebook/zstd/releases/tag/v1.5.6 ) In my opinion, it is better to check the original repository: https://github.com/inikep/lzbench https://github.com/inikep/lzbench or this: https://github.com/inikep/lzbench/blob/master/doc/lzbench20_sorted.md https://github.com/inikep/lzbench/blob/master/doc/lzbench20_...

orthoxerox 2y ago
Judging by the last link, the only options that are strictly superior to snappy in every sense are lz4 (as OP claimed) and zstd (levels -3 to +1). If compression speed is critical, then the default level of zstd compression in most big data tools (+3) might be the wrong one for you, but test on your own data, not Silesia.

gotoeleven 2y ago

Based on these results, why would you use anything but lz4_fast ? Are there other dimensions that might be important for a particular use case so a different algo may be best? Ones I can think of might be multi threaded or SIMD friendliness, cache efficiency, memory efficiency, information locality, stream-ability

pella 2y ago
latest - sorted by Ratio : https://github.com/inikep/lzbench/blob/master/doc/lzbench20_sorted.md https://github.com/inikep/lzbench/blob/master/doc/lzbench20_...
Galanwe 2y ago
> Are there other dimensions that might be important for a particular use case so a different algo may be best? Choosing a compression/decompression algo is an optimization problem essentially. It mainly depends on your IO for input. For instance, if you have very fast IO, say a local SSD at 7GB/s, then you probably don't want compression, as even the fastest decompression will not go beyond 4GB/s. If you have moderately fast IO, say 4GB/s, then lz4 is likely a good candidate, you will get maybe a 0.5 compression ratio, meaning you will spend half the time reading your input, while decompressing at nearly the same same speed that you would read uncompressed data. If you have very slow IO, say 100MB/s, then the compression ratio becomes the main factor, overriding decompression speed. So, essentially, you want to solve an optimization problem of lowering the time to read the compressed file and decompressing it, conditioned by your reading speed, writing speed, and CPU to spare, under a time constraint of compression speed. There are solvers on internet doe this problem, but I would love to see someone write it down here in plain math.

nolist_policy 2y ago

The data sets are way too small, the biggest one (silesia/mozilla) is just 50mb.