3 ms·

zstd hardly replaces xz, the compression ratio is quite worse. zstd seem more of a replacement for gz.

by ars 4y ago

metadat 4y ago
Do you have any data to support this claim? In my experience, zstd is way better in every way compared to gzip. Additionally, xz is good compression but horribly crazy slow to decompress. Xz also only operates on one single file at a time, which is annoying.
- jmillikin 4y ago
  I just did a quick test with a copy of GNU M4, which is reasonably representative of a source code archive. $ time xz -9k m4-1.4.19.tar real 0m2.928s user 0m2.871s sys 0m0.056s $ time zstd -19 m4-1.4.19.tar real 0m3.411s user 0m3.380s sys 0m0.032s $ ls -l m4-1.4.19.tar* -rw-rw-r-- 1 john john 14837760 Jul 24 14:40 m4-1.4.19.tar -rw-rw-r-- 1 john john 1674612 Jul 24 14:40 m4-1.4.19.tar.xz -rw-rw-r-- 1 john john 1726155 Jul 24 14:40 m4-1.4.19.tar.zst In this test, XZ was both faster and had better compression than Zstd.
  - metadat 4y ago
    Howdy jmillikin. That is a very small file. See: https://news.ycombinator.com/item?id=25455846#:~:text=xz%20on%20highest%20(normal)%20compression,speeds%20regardless%20of%20compression%20ratio https://news.ycombinator.com/item?id=25455846#:~:text=xz%20o.... > xz on highest (normal) compression level easily beats the compression ratio of zstd on highest compression level (--ultra -22) on any data I've tested. However with xz reading the compressed files easily becomes a bottleneck, zstd has great read speeds regardless of compression ratio
    - jmillikin 4y ago
      Are you agreeing or disagreeing with ars' claim that XZ provides a better compression ratio than Zstd? My data shows that it's true in at least one common use case (distribution of open-source software source archives). I've seen similar comparative ratios from files up to the multi-gigabyte range, for example VM images. In what cases have you seen XZ produce worse compression ratios than Zstd?
      - JoshTriplett 4y ago
        Generally speaking, the top-end of xz very slightly beats the top-end of zstd. However, xz typically takes several times as long to extract. And generally I've seen xz take longer to compress than zstd, as well. Example with a large archive (representative of compiled software distribution, such as package management formats): $ time xz -T0 -9k usrbin.tar real 2m0.579s user 8m46.646s sys 0m2.104s $ time zstd -T0 -19 --long usrbin.tar real 1m47.242s user 6m34.845s sys 0m0.544s /tmp$ ls -l usrbin.tar* -rw-r--r-- 1 josh josh 998830080 Jul 23 23:55 usrbin.tar -rw-r--r-- 1 josh josh 189633464 Jul 23 23:55 usrbin.tar.xz -rw-r--r-- 1 josh josh 203107989 Jul 23 23:55 usrbin.tar.zst /tmp$ time xzcat usrbin.tar.xz >/dev/null real 0m9.410s user 0m9.339s sys 0m0.060s /tmp$ time zstdcat usrbin.tar.zst >/dev/null real 0m0.996s user 0m0.894s sys 0m0.065s Comparable compression ratio, faster to compress, 10x faster to decompress. And if you do need a smaller compression ratio than xz, you can get that at a cost in time: $ time zstd -T0 -22 --ultra --long usrbin.tar real 4m32.056s user 9m2.484s sys 0m0.644s $ ls -l usrbin.tar* -rw-r--r-- 1 josh josh 998830080 Jul 23 23:55 usrbin.tar -rw-r--r-- 1 josh josh 189633464 Jul 23 23:55 usrbin.tar.xz -rw-r--r-- 1 josh josh 186113543 Jul 23 23:55 usrbin.tar.zst And it still takes the same amount of time to extract, 10x faster than xz.
        jmillikin 4y ago
        That seems fine -- it's a tradeoff between speed and compression ratio, which has existed ever since compression went beyond RLE. Zstd competes against Snappy and LZ4 in the market of transmission-time compression. You use it for things like RPC sessions, where the data is being created on-the-fly, compressed for bandwidth savings, then decompressed+parsed on the other side. And in this domain, Zstd is pretty clearly the stand-out winner. When it comes to archival, the wall-clock performance is less important. Doubling the compress/decompress time for a 5% improvement in compression ratio is an attractive option, and high-compression XZ is in many cases faster than high-compression Zstd even delivering better ratios. --- EDIT for parent post adding numbers: I spot-tested running zstd with `-22 --ultra` on files in my archive of source tarballs, and wasn't able to find cases where it outperformed `xz -9`.
      - 4y ago
klodolph 4y ago
Sure, if you look at the Pareto frontier for xz and zstd, zstd does not seem like a “replacement” for xz. It’s not a replacement for PPMd. The problem is that xz has kind of horrible performance when you crank it up to the high settings. On the medium settings, you can get the same ratio for much, much less CPU (round-trip) by switching to zstd. YMMV, use your own corpus and CPU to test it.
basilgohar 4y ago
Most tests I've seen, such as [0] don't support your statement. Zstd can compress almost as well as xz but decompresses much faster. It can also compress more than xz with tweaks, though I don't know the compute/memory tradeoffs. [0] https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/ https://archlinux.org/news/now-using-zstandard-instead-of-xz... Edit to fix typo.
usr1106 4y ago
Depends on your use case. Saving the last couple of Bytes is often less important than fast compression.
svnpenn 4y ago
Yes, because ratio is the only thing that matters. Decompression speed doesn't matter at all. Who cares if Zstd is 9 times faster?

zstd hardly replaces xz, the compression ratio is quite worse. zstd seem more of a replacement for gz.

by ars 4y ago

zstd hardly replaces xz, the compression ratio is quite worse. zstd seem more of a replacement for gz.

Do you have any data to support this claim? In my experience, zstd is way better in every way compared to gzip. Additionally, xz is good compression but horribly crazy slow to decompress. Xz also only operates on one single file at a time, which is annoying.

jmillikin 4y ago
I just did a quick test with a copy of GNU M4, which is reasonably representative of a source code archive. $ time xz -9k m4-1.4.19.tar real 0m2.928s user 0m2.871s sys 0m0.056s $ time zstd -19 m4-1.4.19.tar real 0m3.411s user 0m3.380s sys 0m0.032s $ ls -l m4-1.4.19.tar* -rw-rw-r-- 1 john john 14837760 Jul 24 14:40 m4-1.4.19.tar -rw-rw-r-- 1 john john 1674612 Jul 24 14:40 m4-1.4.19.tar.xz -rw-rw-r-- 1 john john 1726155 Jul 24 14:40 m4-1.4.19.tar.zst In this test, XZ was both faster and had better compression than Zstd.
- metadat 4y ago
  Howdy jmillikin. That is a very small file. See: https://news.ycombinator.com/item?id=25455846#:~:text=xz%20on%20highest%20(normal)%20compression,speeds%20regardless%20of%20compression%20ratio https://news.ycombinator.com/item?id=25455846#:~:text=xz%20o.... > xz on highest (normal) compression level easily beats the compression ratio of zstd on highest compression level (--ultra -22) on any data I've tested. However with xz reading the compressed files easily becomes a bottleneck, zstd has great read speeds regardless of compression ratio
  - jmillikin 4y ago
    Are you agreeing or disagreeing with ars' claim that XZ provides a better compression ratio than Zstd? My data shows that it's true in at least one common use case (distribution of open-source software source archives). I've seen similar comparative ratios from files up to the multi-gigabyte range, for example VM images. In what cases have you seen XZ produce worse compression ratios than Zstd?
    - JoshTriplett 4y ago
      Generally speaking, the top-end of xz very slightly beats the top-end of zstd. However, xz typically takes several times as long to extract. And generally I've seen xz take longer to compress than zstd, as well. Example with a large archive (representative of compiled software distribution, such as package management formats): $ time xz -T0 -9k usrbin.tar real 2m0.579s user 8m46.646s sys 0m2.104s $ time zstd -T0 -19 --long usrbin.tar real 1m47.242s user 6m34.845s sys 0m0.544s /tmp$ ls -l usrbin.tar* -rw-r--r-- 1 josh josh 998830080 Jul 23 23:55 usrbin.tar -rw-r--r-- 1 josh josh 189633464 Jul 23 23:55 usrbin.tar.xz -rw-r--r-- 1 josh josh 203107989 Jul 23 23:55 usrbin.tar.zst /tmp$ time xzcat usrbin.tar.xz >/dev/null real 0m9.410s user 0m9.339s sys 0m0.060s /tmp$ time zstdcat usrbin.tar.zst >/dev/null real 0m0.996s user 0m0.894s sys 0m0.065s Comparable compression ratio, faster to compress, 10x faster to decompress. And if you do need a smaller compression ratio than xz, you can get that at a cost in time: $ time zstd -T0 -22 --ultra --long usrbin.tar real 4m32.056s user 9m2.484s sys 0m0.644s $ ls -l usrbin.tar* -rw-r--r-- 1 josh josh 998830080 Jul 23 23:55 usrbin.tar -rw-r--r-- 1 josh josh 189633464 Jul 23 23:55 usrbin.tar.xz -rw-r--r-- 1 josh josh 186113543 Jul 23 23:55 usrbin.tar.zst And it still takes the same amount of time to extract, 10x faster than xz.
      - jmillikin 4y ago
        That seems fine -- it's a tradeoff between speed and compression ratio, which has existed ever since compression went beyond RLE. Zstd competes against Snappy and LZ4 in the market of transmission-time compression. You use it for things like RPC sessions, where the data is being created on-the-fly, compressed for bandwidth savings, then decompressed+parsed on the other side. And in this domain, Zstd is pretty clearly the stand-out winner. When it comes to archival, the wall-clock performance is less important. Doubling the compress/decompress time for a 5% improvement in compression ratio is an attractive option, and high-compression XZ is in many cases faster than high-compression Zstd even delivering better ratios. --- EDIT for parent post adding numbers: I spot-tested running zstd with `-22 --ultra` on files in my archive of source tarballs, and wasn't able to find cases where it outperformed `xz -9`.
    - 4y ago

klodolph 4y ago

Sure, if you look at the Pareto frontier for xz and zstd, zstd does not seem like a “replacement” for xz. It’s not a replacement for PPMd. The problem is that xz has kind of horrible performance when you crank it up to the high settings. On the medium settings, you can get the same ratio for much, much less CPU (round-trip) by switching to zstd. YMMV, use your own corpus and CPU to test it.

basilgohar 4y ago

Most tests I've seen, such as [0] don't support your statement. Zstd can compress almost as well as xz but decompresses much faster. It can also compress more than xz with tweaks, though I don't know the compute/memory tradeoffs. [0] https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/ https://archlinux.org/news/now-using-zstandard-instead-of-xz... Edit to fix typo.

usr1106 4y ago

Depends on your use case. Saving the last couple of Bytes is often less important than fast compression.

svnpenn 4y ago

Yes, because ratio is the only thing that matters. Decompression speed doesn't matter at all. Who cares if Zstd is 9 times faster?