4 ms·
Standard comment on zstd: The "zstdcat" program (equivalent to "zstd -cdfq", I think) is capable of reading a few other compression formats, including .gz and .
by waterhouse 8y ago
Standard comment on zstd: The "zstdcat" program (equivalent to "zstd -cdfq", I think) is capable of reading a few other compression formats, including .gz and .lz4 (apparently .lzma and .xz are available too), if support is built in at compile time. Unbelievably, zstdcat is 1.5x faster (in my experience) at decoding .gz files than zcat / gunzip / gzip -d. I dare you to try it.
Anyway, this should ease transitions away from legacy compression formats.
- stochastic_monk 8y agoIt’s also quite convenient supporting a variety of compressed formats with one API.
- goodside 8y agoIf you find 1.5x-faster gzip useful, you should consider benchmarking pigz, a parallel gzip implementation that goes faster still: https://zlib.net/pigz/ https://zlib.net/pigz/ Similarly there's lbzip2 ( http://lbzip2.org http://lbzip2.org ) for parallel bz2.
- waterhouse 8y agoGood point. But for my use case, I have hundreds or more of logfiles, and I parallelize at that level already. So I'm concerned with the total CPU time per file more than anything else.