3 ms·
For system backups and logs and stuff that rarely need to be decompressed, but stored for a while and then deleted in rotation, I still strongly prefer xzip. It
by walrus01 6y ago
For system backups and logs and stuff that rarely need to be decompressed, but stored for a while and then deleted in rotation, I still strongly prefer xzip. It's much slower to create a .tar.xz file with the default settings, but also saves a considerable amount of disk space.
Side by side comparison right now between a full system file tree backup of a mediawiki server, the .tar.xz file is about 1.8GB, the .tar.zstd file is 2.15GB. The same thing in traditional .tar.gz format is obviously even larger.
From the research I've done on common uses of zstd it's much better suited to applications where very high speed of compression and decompression is a major requirement, but absolute smallest file size is not. Typically as a replacement for gzip.
- petre 6y agoTry xz -1. Faster than gzip, smaller size. We use it for regular backups. Although the xz format is not recommended for long term archiving because it's unsafe. https://www.nongnu.org/lzip/xz_inadequate.html https://www.nongnu.org/lzip/xz_inadequate.html
- walrus01 6y agoI'm going to read this in more detail later today when I have time. But I just skimmed it, and most of the points about xz being fragile and having a weird format would seem to apply in a scenario in which a .tar.xz file became corrupted or bit-flipped from its original? eg: in which the .xz file, after some long period of archiving, no longer matched the same sha256sum as when it was originally created. Thereby making it difficult or a huge hassle to get authentic lossless/uncorrupted data out of it?
- loeg 6y agoThe "unsafe" thing is nonsense. The expressed concern is how the contents degrade in the presence of arbitrary bit flips. Any highly-compressed format is going to be susceptible to corruption amplification from random bitflips, and the higher compression level, the larger fallout. Ensure your compressed archives are not corrupted using some external FEC; either in your hardware, or something like par2.
- The_rationalist 6y agoThose are with the default settings but can't you increase the compression ratio target of zstd to match xz and still be significantly faster?
- walrus01 6y agoIn my tests with the mix of files on this particular virtual machine, increasing the compression ratio of zstd to achieve an equally small output file size as xzip, it's equally as slow on a per-core basis. It does have a performance advantage in that it by default uses both CPU cores, while a default tar with -J option will use xzip on only one CPU core.
- loeg 6y agoIf the compression ratio is about the same, I'd choose zstd. The decompression performance is much better than xz.
- The_rationalist 6y agoAnd zstd keeps evolving contrary to xz, which means they'll outperforms xz compression ratio in the following years
- loeg 6y agoWe can hope!
- wongarsu 6y agoxz on highest (normal) compression level easily beats the compression ratio of zstd on highest compression level (--ultra -22) on any data I've tested. However with xz reading the compressed files easily becomes a bottleneck, zstd has great read speeds regardless of compression ratio.
- loeg 6y agoIn my experience (a couple years ago, YMMV) high-level (22) zstd compression is about as slow as high-level xz (9) compression and the results are slightly larger. High-level zstd decompresses vastly faster than xz — so I would use it for things like CDNs, where users are decompressing contents much more frequently than compression — but for backups, which are decompressed zero or one times, it probably doesn't matter.