4 ms·

> But most people I work with will pick zstd every time even in cases where decompression speed matters the most. Zstd is really good. If you're a math/compre

by loxias 4y ago

> But most people I work with will pick zstd every time even in cases where decompression speed matters the most. Zstd is really good. If you're a math/compression theory nerd it makes a lot of good choices. Also, if you're willing to tune zstd instead of using the defaults, so far anyway, I haven't found any application where the right answer wasn't zstd. At this point I don't think it's unreasonable to expect new compression developments to to explicitly say "What is improved over ZStandard".

ciupicri 4y ago
Just to add to what you're saying, you can't tune zstd for qcow images or for btrfs filesystems.
- loxias 4y ago
  You're entirely right! And points out the blind spot that I've used it much more as a library in tools I build/use than integrated with other things. Never used brtfs, I'm assuming it uses zstd internally for something? Cool. Also very cool that qcow contains zstd support (didn't know that either!).
  - ciupicri 4y ago
    From https://btrfs.wiki.kernel.org/index.php/Compression https://btrfs.wiki.kernel.org/index.php/Compression > Btrfs supports transparent file compression. There are three algorithms available: ZLIB, LZO and ZSTD(since v4.14) I've been using zstd compression for qcow2 images since last year, but I admit it's kind of hidden. The man page for qemu-img [1] doesn't say a word about zstd, but it's there alright [2]. [1]: https://www.qemu.org/docs/master/tools/qemu-img.html https://www.qemu.org/docs/master/tools/qemu-img.html [2]: https://gitlab.com/qemu-project/qemu/-/blob/v7.0.0/block/qcow2-threads.c#L188 https://gitlab.com/qemu-project/qemu/-/blob/v7.0.0/block/qco...
CoolGuySteve 4y ago
That's the thing, even at its fastest setting, zstd is still slower than lz4. So clearly, it's not the right answer if you're serializing from a fast medium. And this is exactly the kind of cognitive laziness I'm talking about.
- arinlen 4y ago
  > That's the thing, even at its fastest setting, zstd is still slower than lz4. So clearly, it's not the right answer if you're serializing from a fast medium. It really depends on what you want to do, as decompression speeds can be meaningless and irrelevant in some applications, if not practically all of them. Can you provide a specific example where you feel lz4 is hands down the best option?
  - CoolGuySteve 4y ago
    Yes, I can and did. When serializing off a fast medium like nvme drives. And again, I shouldn’t have to keep explaining this. zstd isn’t some magic answer to everything.
    - Dylan16807 4y ago
      > I shouldn’t have to keep explaining this. zstd isn’t some magic answer to everything. That logic doesn't really work. It's true that zstd doesn't let you tune it to be as fast as lz4. But it's possible that a slightly different design could make that possible, so that the new version is better than lz4 at all speeds and at all ratios. Making one compression algorithm strictly better than another requires no magic.
- adgjlsfhk1 4y ago
  Note that ZSTD decompression is generally faster for data compressed with slower settings.
  - deepsun 4y ago
    Nope, as I wrote above, ZSTD decompression is always slower than Lizard's, regardless of settings.

by loxias 4y ago

ciupicri 4y ago

Just to add to what you're saying, you can't tune zstd for qcow images or for btrfs filesystems.

loxias 4y ago
You're entirely right! And points out the blind spot that I've used it much more as a library in tools I build/use than integrated with other things. Never used brtfs, I'm assuming it uses zstd internally for something? Cool. Also very cool that qcow contains zstd support (didn't know that either!).
- ciupicri 4y ago
  From https://btrfs.wiki.kernel.org/index.php/Compression https://btrfs.wiki.kernel.org/index.php/Compression > Btrfs supports transparent file compression. There are three algorithms available: ZLIB, LZO and ZSTD(since v4.14) I've been using zstd compression for qcow2 images since last year, but I admit it's kind of hidden. The man page for qemu-img [1] doesn't say a word about zstd, but it's there alright [2]. [1]: https://www.qemu.org/docs/master/tools/qemu-img.html https://www.qemu.org/docs/master/tools/qemu-img.html [2]: https://gitlab.com/qemu-project/qemu/-/blob/v7.0.0/block/qcow2-threads.c#L188 https://gitlab.com/qemu-project/qemu/-/blob/v7.0.0/block/qco...

CoolGuySteve 4y ago

That's the thing, even at its fastest setting, zstd is still slower than lz4. So clearly, it's not the right answer if you're serializing from a fast medium. And this is exactly the kind of cognitive laziness I'm talking about.

arinlen 4y ago
> That's the thing, even at its fastest setting, zstd is still slower than lz4. So clearly, it's not the right answer if you're serializing from a fast medium. It really depends on what you want to do, as decompression speeds can be meaningless and irrelevant in some applications, if not practically all of them. Can you provide a specific example where you feel lz4 is hands down the best option?
- CoolGuySteve 4y ago
  Yes, I can and did. When serializing off a fast medium like nvme drives. And again, I shouldn’t have to keep explaining this. zstd isn’t some magic answer to everything.
  - Dylan16807 4y ago
    > I shouldn’t have to keep explaining this. zstd isn’t some magic answer to everything. That logic doesn't really work. It's true that zstd doesn't let you tune it to be as fast as lz4. But it's possible that a slightly different design could make that possible, so that the new version is better than lz4 at all speeds and at all ratios. Making one compression algorithm strictly better than another requires no magic.
adgjlsfhk1 4y ago
Note that ZSTD decompression is generally faster for data compressed with slower settings.
- deepsun 4y ago
  Nope, as I wrote above, ZSTD decompression is always slower than Lizard's, regardless of settings.