4 ms·

Those are with the default settings but can't you increase the compression ratio target of zstd to match xz and still be significantly faster?

walrus01 6y ago
In my tests with the mix of files on this particular virtual machine, increasing the compression ratio of zstd to achieve an equally small output file size as xzip, it's equally as slow on a per-core basis. It does have a performance advantage in that it by default uses both CPU cores, while a default tar with -J option will use xzip on only one CPU core.
- loeg 6y ago
  If the compression ratio is about the same, I'd choose zstd. The decompression performance is much better than xz.
  - The_rationalist 6y ago
    And zstd keeps evolving contrary to xz, which means they'll outperforms xz compression ratio in the following years
    - loeg 6y ago
      We can hope!
wongarsu 6y ago
xz on highest (normal) compression level easily beats the compression ratio of zstd on highest compression level (--ultra -22) on any data I've tested. However with xz reading the compressed files easily becomes a bottleneck, zstd has great read speeds regardless of compression ratio.
loeg 6y ago
In my experience (a couple years ago, YMMV) high-level (22) zstd compression is about as slow as high-level xz (9) compression and the results are slightly larger. High-level zstd decompresses vastly faster than xz — so I would use it for things like CDNs, where users are decompressing contents much more frequently than compression — but for backups, which are decompressed zero or one times, it probably doesn't matter.
- iruoy 6y ago
  This is basically what arch linux has done with their repo/packages. > zstd and xz trade blows in their compression ratio. Recompressing all packages to zstd with our options yields a total ~0.8% increase in package size on all of our packages combined, but the decompression time for all packages saw a ~1300% speedup.
  - walrus01 6y ago
    In some things it's often a trade off of expected network speed vs cpu power. On very limited network speed connections such as a last mile ISP connection in a developing nation, where speeds might be under 5Mbps downstream, xz has some advantages in lower transfer times. Often times persons on very limited bandwidth connections (even a Hughesnet or Viasat consumer grade satellite link in a very rural area of the USA) will have more than sufficient CPU to decompress xz in a reasonable amount of time. Such as a five year old core i5 desktop computer. In environments where it can be expected that the bulk of users are on very fast broadband connections, zstd and slightly larger file sizes have an advantage.
    - loeg 6y ago
      This (purported) benefit is bounded by the difference between compression ratios. You're responding to a claim that the difference was 0.8%. This is pretty marginal — that first 99.2% takes exactly the same with xz or zstd, and the remaining 0.8% is not large in comparison. Take for example ~13.5 GB. This is a 6 hour download at your 5Mbps figure. An extra 0.8% is less than 3 minutes on that 6 hour download.

walrus01 6y ago

In my tests with the mix of files on this particular virtual machine, increasing the compression ratio of zstd to achieve an equally small output file size as xzip, it's equally as slow on a per-core basis. It does have a performance advantage in that it by default uses both CPU cores, while a default tar with -J option will use xzip on only one CPU core.

loeg 6y ago
If the compression ratio is about the same, I'd choose zstd. The decompression performance is much better than xz.
- The_rationalist 6y ago
  And zstd keeps evolving contrary to xz, which means they'll outperforms xz compression ratio in the following years
  - loeg 6y ago
    We can hope!

wongarsu 6y ago

xz on highest (normal) compression level easily beats the compression ratio of zstd on highest compression level (--ultra -22) on any data I've tested. However with xz reading the compressed files easily becomes a bottleneck, zstd has great read speeds regardless of compression ratio.

loeg 6y ago

In my experience (a couple years ago, YMMV) high-level (22) zstd compression is about as slow as high-level xz (9) compression and the results are slightly larger. High-level zstd decompresses vastly faster than xz — so I would use it for things like CDNs, where users are decompressing contents much more frequently than compression — but for backups, which are decompressed zero or one times, it probably doesn't matter.

iruoy 6y ago
This is basically what arch linux has done with their repo/packages. > zstd and xz trade blows in their compression ratio. Recompressing all packages to zstd with our options yields a total ~0.8% increase in package size on all of our packages combined, but the decompression time for all packages saw a ~1300% speedup.
- walrus01 6y ago
  In some things it's often a trade off of expected network speed vs cpu power. On very limited network speed connections such as a last mile ISP connection in a developing nation, where speeds might be under 5Mbps downstream, xz has some advantages in lower transfer times. Often times persons on very limited bandwidth connections (even a Hughesnet or Viasat consumer grade satellite link in a very rural area of the USA) will have more than sufficient CPU to decompress xz in a reasonable amount of time. Such as a five year old core i5 desktop computer. In environments where it can be expected that the bulk of users are on very fast broadband connections, zstd and slightly larger file sizes have an advantage.
  - loeg 6y ago
    This (purported) benefit is bounded by the difference between compression ratios. You're responding to a claim that the difference was 0.8%. This is pretty marginal — that first 99.2% takes exactly the same with xz or zstd, and the remaining 0.8% is not large in comparison. Take for example ~13.5 GB. This is a 6 hour download at your 5Mbps figure. An extra 0.8% is less than 3 minutes on that 6 hour download.