3 ms·
In the gzip case, the reason is that the compressed stream format is inherently sequential: you need to decompress everything before a byte before you can decom
by bennofs 6y ago
In the gzip case, the reason is that the compressed stream format is inherently sequential: you need to decompress everything before a byte before you can decompress the next byte. So parallizing is not really possible, without changing the format.
Newer compression formats have multiple streams that can be decompressed in parallel, or have blocks that are compressed independently without dependencies (bzip2 can be decompressed in parallel due to independent blocks).
- ed25519FUUU 6y agoSo is Pigz not backward compatible with gzip?
- trav4225 6y agoI don't think Pigz decompresses in parallel.
- goodside 6y agoPigz is backward compatible — it breaks the input into 128K chunks, compresses each in parallel, and concatenates them. Gzip supports naive concatenation of compressed files so the result can be decompressed by standard gzip. Pigz does not implement parallel decompression, but pigz-compressed files could, in principle, be decompressed in parallel. I assume there's not much urgency for this because gzip decompression is fast enough: >3x faster than compression and >10x faster than bzip2 decompression on default settings. (See: https://quixdb.github.io/squash-benchmark/ https://quixdb.github.io/squash-benchmark/ )