4 ms·
That's part of the benefit of zstd, as I understand it; near-xz compression ratios but much faster.
by yjftsjthsd-h 7y ago
That's part of the benefit of zstd, as I understand it; near-xz compression ratios but much faster.
- loeg 7y agoCompression is only mildly faster or on par with xz, but decompression is (at similar ratios) vastly faster. Which really helps consumers of compressed blobs.
- yjftsjthsd-h 7y agoYeah, that's a useful distinction. So it's excellent for ex. packages (hence https://www.archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/ https://www.archlinux.org/news/now-using-zstandard-instead-o...), but iffy for write-once-read-hopefully-never backups. (Although I've heard it suggested that this might flip again the moment you start test-restoring backups regularly)
- loeg 7y agoAgreed. I think I'd take zstd over xz for WORN backups anyway, just because it's pretty reliable at detecting stream corruption. (Then again, I suggest generating par2 or other FEC against your compressed backups so that's not a problem.)
- terrelln 7y agoZstd has a 32-bit checksum over the uncompressed data, which is enabled by default on the CLI.
- loeg 7y ago(I know you know this, as one of the principals; this is for other readers.) And importantly, that 32-bit checksum is a pretty good checksum; a truncated XXH64(): https://tools.ietf.org/html/rfc8478#section-3 https://tools.ietf.org/html/rfc8478#section-3 That's about as high-quality as you can expect from 32 bits. It's not, say, Fletcher-32.
- JyrkiAlakuijala 7y agoTwo good options here: Brotli gets within 0.6 % of the density of lzma, zstd within 5 %. Brotli is 5x faster than lzma and zstd 9x. I'm somewhat surprised that some people migrated from lzma to zstd, without considering brotli.