4 ms·

This is basically what arch linux has done with their repo/packages. > zstd and xz trade blows in their compression ratio. Recompressing all packages to zstd w

by iruoy 6y ago

This is basically what arch linux has done with their repo/packages. > zstd and xz trade blows in their compression ratio. Recompressing all packages to zstd with our options yields a total ~0.8% increase in package size on all of our packages combined, but the decompression time for all packages saw a ~1300% speedup.

walrus01 6y ago
In some things it's often a trade off of expected network speed vs cpu power. On very limited network speed connections such as a last mile ISP connection in a developing nation, where speeds might be under 5Mbps downstream, xz has some advantages in lower transfer times. Often times persons on very limited bandwidth connections (even a Hughesnet or Viasat consumer grade satellite link in a very rural area of the USA) will have more than sufficient CPU to decompress xz in a reasonable amount of time. Such as a five year old core i5 desktop computer. In environments where it can be expected that the bulk of users are on very fast broadband connections, zstd and slightly larger file sizes have an advantage.
- loeg 6y ago
  This (purported) benefit is bounded by the difference between compression ratios. You're responding to a claim that the difference was 0.8%. This is pretty marginal — that first 99.2% takes exactly the same with xz or zstd, and the remaining 0.8% is not large in comparison. Take for example ~13.5 GB. This is a 6 hour download at your 5Mbps figure. An extra 0.8% is less than 3 minutes on that 6 hour download.
  - walrus01 6y ago
    For the corpus of things I've dealt with recently, the difference is a lot more than 0.8% in size. As much as the difference between an entire compressed virtual machine (copy of a set of LVM logical volumes that have been copied with dd and compressed) being 4.2GB vs 5.15GB in size. If it might take ten hours to transfer 900MB in some unusual environments, that's a big savings. Also there is a degree of asymmetry, in that it's not so difficult for the system creating the xzip archive to have a lot of CPU (four CPU cores out of 12 or 16 on a dual socket current generation xeon system, or a threadripper). Since xzip is much more time consuming to create than to extract. Or for any other environment where WAN connectivity is extremely limited in throughput.
    - loeg 6y ago
      I'd like to see some real example where the compression difference between xz and zstd is actually 4.2 vs 5.15 GB. That's huge and bigger than any difference I've ever seen published between the two. Are you sure you actually used comparable settings with zstd? IMO, this is not a typical difference and shouldn't guide intuitions about where to use one or the other.

by iruoy 6y ago

walrus01 6y ago

In some things it's often a trade off of expected network speed vs cpu power. On very limited network speed connections such as a last mile ISP connection in a developing nation, where speeds might be under 5Mbps downstream, xz has some advantages in lower transfer times. Often times persons on very limited bandwidth connections (even a Hughesnet or Viasat consumer grade satellite link in a very rural area of the USA) will have more than sufficient CPU to decompress xz in a reasonable amount of time. Such as a five year old core i5 desktop computer. In environments where it can be expected that the bulk of users are on very fast broadband connections, zstd and slightly larger file sizes have an advantage.

loeg 6y ago
This (purported) benefit is bounded by the difference between compression ratios. You're responding to a claim that the difference was 0.8%. This is pretty marginal — that first 99.2% takes exactly the same with xz or zstd, and the remaining 0.8% is not large in comparison. Take for example ~13.5 GB. This is a 6 hour download at your 5Mbps figure. An extra 0.8% is less than 3 minutes on that 6 hour download.
- walrus01 6y ago
  For the corpus of things I've dealt with recently, the difference is a lot more than 0.8% in size. As much as the difference between an entire compressed virtual machine (copy of a set of LVM logical volumes that have been copied with dd and compressed) being 4.2GB vs 5.15GB in size. If it might take ten hours to transfer 900MB in some unusual environments, that's a big savings. Also there is a degree of asymmetry, in that it's not so difficult for the system creating the xzip archive to have a lot of CPU (four CPU cores out of 12 or 16 on a dual socket current generation xeon system, or a threadripper). Since xzip is much more time consuming to create than to extract. Or for any other environment where WAN connectivity is extremely limited in throughput.
  - loeg 6y ago
    I'd like to see some real example where the compression difference between xz and zstd is actually 4.2 vs 5.15 GB. That's huge and bigger than any difference I've ever seen published between the two. Are you sure you actually used comparable settings with zstd? IMO, this is not a typical difference and shouldn't guide intuitions about where to use one or the other.