3 ms·
/usr/bin as input tar czf: 18s (214MB) tar cf: 240ms (553MB) tar c | lz4 -c: 1s (305MB) tar c | zstd -c: 1.6s (214MB) TL;DR gzip is and has been for a
by jeffbee 4y ago
/usr/bin as input
tar czf: 18s (214MB)
tar cf: 240ms (553MB)
tar c | lz4 -c: 1s (305MB)
tar c | zstd -c: 1.6s (214MB)
TL;DR gzip is and has been for a long time obsolete as both a format and as a tool.
find | cpio: 1.2s
Bit disappointed in that last bit, honestly.
- nrclark 4y agoGzip is fine for lots of tasks. It's not the most efficient compressor, either by space or speed. But it has a million implementations that all work well together. If you just need "good enough" compression with no fuss, gzip is still an excellent choice.
- ericbarrett 4y agoI agree with this, even as a long-time zstd user both personally and professionally. The fact is nearly every app, widget, API, and even CPU can encode and decode gzip. Sometimes you need to make the prosaic choice.
- tiagod 4y agoI feel like zstd, if nothing better shows up, will become a lot like gzip in the long run
- somat 4y agofor a long time the status quo was that while there were better compressions engines (bzip2 comes to mind) none were faster than gzip. I actually preferred gzip for a lot of tasks because it was faster to compress and decompress. I saw a comment that zstd is actually faster than gzip so I guess that consideration no longer holds.
- jeffbee 4y agoMost things are faster than gzip, and many of those things have smaller outputs as well. Brotli at level 2 produces the same-sized archive with this input, in just 3.4s. Gzip even at level 1, with a 10% larger archive, still needs 6.5s. zstd, at level 1, produces the same size as gzip -1, in 1 second! zstd, cranked up to 9, makes a 20% smaller archive in 7 seconds, less than half the time of gzip -6.
- ericpauley 4y agoAgreed. I’ve switched everything over to zstd and find using it actually speeds most tasks up (lower disk IO).
- koolba 4y agoDid you run that more than once to ensure uniform cache hits? Otherwise only the first one will pay the price for the actual disk reads.
- glandium 4y agoRecent GNU tar versions have a --zstd flag, FWIW.
- chungy 4y agoIt also has an "-a" flag that just runs the right compressor based on output file name. "tar -acf archive.tar.gz" and "tar -acf archive.tar.zst" will do the right thing in both cases.