3 ms·
gzip is a particularly poor performer when it comes to spotting repetition across larger amounts of data. Use bzip2 instead if you want to give home-baked tarba
by daneel_w 6y ago
gzip is a particularly poor performer when it comes to spotting repetition across larger amounts of data. Use bzip2 instead if you want to give home-baked tarballs a somewhat fair competition against Bupstash etc.
- andrewchambers 6y agoThanks for the suggestion, I mainly used gzip as that is what my muscle memory goes for when compressing tarballs, it is a fair criticism.
- smichel17 6y agoHow does that compare to xz?
- brnt 6y agoOr zstd?
- Aachen 6y agoWant zstd more focused on performance without too big a decrease in compression ratio, rather than getting better compression ratios? I might be mistaken.
- aidenn0 6y agozstd's performance curve is entirely within gzip's. For every level of gzip compression, zstd has a level with better compression at the same speed, and a level with the same (or better) compression at a higher speed. Note that this is all for single core; zstd can also run in parallel.
- Aachen 6y agoThat's really cool, thanks for the info!
- Aachen 6y agoIn my experience it depends on the type of data, though never have I seen it do better or worse than a negligible amount (on real world data, I'm sure you could find the type it's better at and exploit that).
- daneel_w 6y agoxz for the most part does a much better job than bzip2.
- garmaine 6y agoOr lzop for speed, while still being better than gzip.
- garaetjjte 6y agoFor speed, LZ4 is great.
- petre 6y agoI found out that xz -1 produces smaller files than gzip -6, faster. That's what we use now.
- wolf550e 6y agoYou should try zstd, it often beats xz. xz beats zstd only in high compression scenarios (-9).
- deleted 6y ago[deleted]
- rzzzt 6y agoThere's also lrzip for large files: https://github.com/ckolivas/lrzip https://github.com/ckolivas/lrzip
- wolf550e 6y agoBoth gzip and bzip2 are very old. You should use zstd.
- daneel_w 6y agoOr xz, my current favorite.