3 ms·

I just had a little look into these somewhat new zstd options --fast 1 to fast 5. fast 5 does seem almost equivalent to lz4 performance in compression ratio and

I just had a little look into these somewhat new zstd options --fast 1 to fast 5. fast 5 does seem almost equivalent to lz4 performance in compression ratio and compression 'speed' (or cpu load) except decompression 'speed' is still almost half of lz4, so at that extreme of zstds options it is squarely beaten by lz4. But in terms of compression ratio - which in most cases should be the more crucial characteristic for any compressor than cpu load - this setting seems to be well under zstd's knee of diminishing returns. --fast 1 has about 15% more cpu load for about 112% of compression performance than --fast5. And then zstds lightest non-fast setting (zstd-1) for 50% more load in compression and decompression, compresses 32% better than --fast5 (and lzo and lz4). The decompression load is an eye catching feature of lz4, but its compression ratio is not just marginally compromised, it is a much less powerful compressor in that prime characteristic than zstds weakest non-fast setting. I can appreciate why Btrfs is in no hurry to include lz4 while it supports zstd:1 to 15. Although it would be interesting to test if lz4 could have a niche on fast storage combined with slow cpus.

ahartmetz 5y ago
There are use cases for lz4, I've had one in an embedded project. Slowish SoC (i.MX6 DualLite IIRC), relatively fast storage (eMMC - these tend to perform better than regular MMC). I measured boot time for the available kernel compression options and lz4 won by 100-200 ms or so compared to the next best option, which may have been gzip. It didn't make a big dent in overall boot time, but not bad for just setting an option. zstd, if available, which it wasn't, would have gotten a marginally worse result probably. IIRC, lzma was slower than uncompressed.
- strainer 5y ago
  gzip would seem to be a surprising runner up as its decompression cpu load is reportedly around 3-4 times as heavy as zstd:1 with a similar compression ratio. I would not rule zstd out, it seems to better every alternative given the right level setting, except lz4 which is the outlier but fixed at a relatively low compression level. As the high performing outlier it tantalisingly suggests room for improvements :]
  - ahartmetz 5y ago
    At least older Linux (the kernel) versions don't support zstd compression (for the kernel image itself), that's why it was not part of the experiment.
    - strainer 5y ago
      I read it was only mainlined last year following this thread[1] where a kernel dev presented benchmarks and advised "zstd is an improvement on almost all metrics" Of course zstd somewhat of a portfolio project of facebook developers, and they seem to have integrated it quite nicely into Btrfs to atone for their debt to society - I'll take that :D [1] https://lore.kernel.org/lkml/1588791882.08g1378g67.none@localhost/T/ https://lore.kernel.org/lkml/1588791882.08g1378g67.none@loca...

ahartmetz 5y ago

There are use cases for lz4, I've had one in an embedded project. Slowish SoC (i.MX6 DualLite IIRC), relatively fast storage (eMMC - these tend to perform better than regular MMC). I measured boot time for the available kernel compression options and lz4 won by 100-200 ms or so compared to the next best option, which may have been gzip. It didn't make a big dent in overall boot time, but not bad for just setting an option. zstd, if available, which it wasn't, would have gotten a marginally worse result probably. IIRC, lzma was slower than uncompressed.

strainer 5y ago
gzip would seem to be a surprising runner up as its decompression cpu load is reportedly around 3-4 times as heavy as zstd:1 with a similar compression ratio. I would not rule zstd out, it seems to better every alternative given the right level setting, except lz4 which is the outlier but fixed at a relatively low compression level. As the high performing outlier it tantalisingly suggests room for improvements :]
- ahartmetz 5y ago
  At least older Linux (the kernel) versions don't support zstd compression (for the kernel image itself), that's why it was not part of the experiment.
  - strainer 5y ago
    I read it was only mainlined last year following this thread[1] where a kernel dev presented benchmarks and advised "zstd is an improvement on almost all metrics" Of course zstd somewhat of a portfolio project of facebook developers, and they seem to have integrated it quite nicely into Btrfs to atone for their debt to society - I'll take that :D [1] https://lore.kernel.org/lkml/1588791882.08g1378g67.none@localhost/T/ https://lore.kernel.org/lkml/1588791882.08g1378g67.none@loca...