4 ms·

The thing I don't get about zstd is that their own github page shows lz4 is faster at both compression and decompression at the cost of some compression ratio:

The thing I don't get about zstd is that their own github page shows lz4 is faster at both compression and decompression at the cost of some compression ratio: https://github.com/facebook/zstd https://github.com/facebook/zstd But most people I work with will pick zstd every time even in cases where decompression speed matters the most.

adgjlsfhk1 4y ago
If you are bottle-necked by reading data (eg over networks or slow storage), zstd will be faster to decompress since you are transferring less data.
- arinlen 4y ago
  > If you are bottle-necked by reading data (eg over networks or slow storage), zstd will be faster to decompress since you are transferring less data. This should be stressed. Benchmarks show zstd's compression ratio being 75% of lz4's. Pushing 25% less data over a network is enough to win any real-world test, not to mention the impact it has on operational costs.
  - woobilicious 4y ago
    I have this pathological example, I have a 30GB journal dump from bcachefs, and compressed it down with lz4 and zstd to a sata SSD, the compression ratios are 6.8% and 3% respectively (and this with default level 3!), it takes 14s to ripgrep through the lz4 version, and 11s with the zstd version. So even at 500MB/s zstd is faster for decompression.
    - adgjlsfhk1 4y ago
      LZ4 requires about 2gb/s to be competitive on any relatively recent desktop/laptop cpu
loxias 4y ago
> But most people I work with will pick zstd every time even in cases where decompression speed matters the most. Zstd is really good. If you're a math/compression theory nerd it makes a lot of good choices. Also, if you're willing to tune zstd instead of using the defaults, so far anyway, I haven't found any application where the right answer wasn't zstd. At this point I don't think it's unreasonable to expect new compression developments to to explicitly say "What is improved over ZStandard".
- ciupicri 4y ago
  Just to add to what you're saying, you can't tune zstd for qcow images or for btrfs filesystems.
  - loxias 4y ago
    You're entirely right! And points out the blind spot that I've used it much more as a library in tools I build/use than integrated with other things. Never used brtfs, I'm assuming it uses zstd internally for something? Cool. Also very cool that qcow contains zstd support (didn't know that either!).
    - ciupicri 4y ago
      From https://btrfs.wiki.kernel.org/index.php/Compression https://btrfs.wiki.kernel.org/index.php/Compression > Btrfs supports transparent file compression. There are three algorithms available: ZLIB, LZO and ZSTD(since v4.14) I've been using zstd compression for qcow2 images since last year, but I admit it's kind of hidden. The man page for qemu-img [1] doesn't say a word about zstd, but it's there alright [2]. [1]: https://www.qemu.org/docs/master/tools/qemu-img.html https://www.qemu.org/docs/master/tools/qemu-img.html [2]: https://gitlab.com/qemu-project/qemu/-/blob/v7.0.0/block/qcow2-threads.c#L188 https://gitlab.com/qemu-project/qemu/-/blob/v7.0.0/block/qco...
- CoolGuySteve 4y ago
  That's the thing, even at its fastest setting, zstd is still slower than lz4. So clearly, it's not the right answer if you're serializing from a fast medium. And this is exactly the kind of cognitive laziness I'm talking about.

adgjlsfhk1 4y ago

If you are bottle-necked by reading data (eg over networks or slow storage), zstd will be faster to decompress since you are transferring less data.

arinlen 4y ago
> If you are bottle-necked by reading data (eg over networks or slow storage), zstd will be faster to decompress since you are transferring less data. This should be stressed. Benchmarks show zstd's compression ratio being 75% of lz4's. Pushing 25% less data over a network is enough to win any real-world test, not to mention the impact it has on operational costs.
- woobilicious 4y ago
  I have this pathological example, I have a 30GB journal dump from bcachefs, and compressed it down with lz4 and zstd to a sata SSD, the compression ratios are 6.8% and 3% respectively (and this with default level 3!), it takes 14s to ripgrep through the lz4 version, and 11s with the zstd version. So even at 500MB/s zstd is faster for decompression.
  - adgjlsfhk1 4y ago
    LZ4 requires about 2gb/s to be competitive on any relatively recent desktop/laptop cpu

loxias 4y ago

> But most people I work with will pick zstd every time even in cases where decompression speed matters the most. Zstd is really good. If you're a math/compression theory nerd it makes a lot of good choices. Also, if you're willing to tune zstd instead of using the defaults, so far anyway, I haven't found any application where the right answer wasn't zstd. At this point I don't think it's unreasonable to expect new compression developments to to explicitly say "What is improved over ZStandard".

ciupicri 4y ago
Just to add to what you're saying, you can't tune zstd for qcow images or for btrfs filesystems.
- loxias 4y ago
  You're entirely right! And points out the blind spot that I've used it much more as a library in tools I build/use than integrated with other things. Never used brtfs, I'm assuming it uses zstd internally for something? Cool. Also very cool that qcow contains zstd support (didn't know that either!).
  - ciupicri 4y ago
    From https://btrfs.wiki.kernel.org/index.php/Compression https://btrfs.wiki.kernel.org/index.php/Compression > Btrfs supports transparent file compression. There are three algorithms available: ZLIB, LZO and ZSTD(since v4.14) I've been using zstd compression for qcow2 images since last year, but I admit it's kind of hidden. The man page for qemu-img [1] doesn't say a word about zstd, but it's there alright [2]. [1]: https://www.qemu.org/docs/master/tools/qemu-img.html https://www.qemu.org/docs/master/tools/qemu-img.html [2]: https://gitlab.com/qemu-project/qemu/-/blob/v7.0.0/block/qcow2-threads.c#L188 https://gitlab.com/qemu-project/qemu/-/blob/v7.0.0/block/qco...
CoolGuySteve 4y ago
That's the thing, even at its fastest setting, zstd is still slower than lz4. So clearly, it's not the right answer if you're serializing from a fast medium. And this is exactly the kind of cognitive laziness I'm talking about.