2 ms·
This (purported) benefit is bounded by the difference between compression ratios. You're responding to a claim that the difference was 0.8%. This is pretty ma
by loeg 6y ago
This (purported) benefit is bounded by the difference between compression ratios. You're responding to a claim that the difference was 0.8%. This is pretty marginal — that first 99.2% takes exactly the same with xz or zstd, and the remaining 0.8% is not large in comparison.
Take for example ~13.5 GB. This is a 6 hour download at your 5Mbps figure. An extra 0.8% is less than 3 minutes on that 6 hour download.
- walrus01 6y agoFor the corpus of things I've dealt with recently, the difference is a lot more than 0.8% in size. As much as the difference between an entire compressed virtual machine (copy of a set of LVM logical volumes that have been copied with dd and compressed) being 4.2GB vs 5.15GB in size. If it might take ten hours to transfer 900MB in some unusual environments, that's a big savings. Also there is a degree of asymmetry, in that it's not so difficult for the system creating the xzip archive to have a lot of CPU (four CPU cores out of 12 or 16 on a dual socket current generation xeon system, or a threadripper). Since xzip is much more time consuming to create than to extract. Or for any other environment where WAN connectivity is extremely limited in throughput.
- loeg 6y agoI'd like to see some real example where the compression difference between xz and zstd is actually 4.2 vs 5.15 GB. That's huge and bigger than any difference I've ever seen published between the two. Are you sure you actually used comparable settings with zstd? IMO, this is not a typical difference and shouldn't guide intuitions about where to use one or the other.