4 ms·
Nice post — one suggestion would be to use zstd instead of gzip though. Out of the box it is much faster and will result in smaller files, and allows tuning to
by ctur 4y ago
Nice post — one suggestion would be to use zstd instead of gzip though. Out of the box it is much faster and will result in smaller files, and allows tuning to fit inside memory and cpu thresholds different use cases might prefer.
Friends don’t let friends use gzip :)
- otterley 4y agoStorage is so cheap nowadays that usability and compression speed are more important for most people.
- usefulcat 4y agoFor a given compression ratio, zstd compression is faster than gzip by a good margin (even without using multiple threads). And zstd decompression is pretty much always faster than gzip too.
- jrockway 4y agoHere is a nice page for mapping "what first came to mind" to "what's actually fastest in 2021": https://jolynch.github.io/posts/use_fast_data_algorithms/ https://jolynch.github.io/posts/use_fast_data_algorithms/ TL;DR, lz4 is 10x faster than gzip (though does not produce as small as a file), zstd is twice as fast and better.
- benbjohnson 4y agoThanks, I thought about using a different compression tool but everybody knows gzip so I figured I’d keep it simple. Litestream uses LZ4 internally (which is great) but it’s not something a lot of developers come across too often.