3 ms·
pigz only parallelizes compression. Decompressing with pigz is single threaded, except perhaps a separate thread is used for crc calculation. A decade ago I im
by mgerdts 4y ago
pigz only parallelizes compression. Decompressing with pigz is single threaded, except perhaps a separate thread is used for crc calculation.
A decade ago I implemented parallel decompresssion for pigz. This is used in Solaris kernel zone suspend and resume, which was the reason I did the work. I submitted a PR for it but madler never got around to reviewing and merging it. Since then there has been a lot of code churn that makes it a pain to apply to the current version.
- fulafel 4y agoDocker is indeed looking for a "unpigz" executable to use: https://github.com/moby/moby/blob/c9d2b7df777b38f7239a882c27c763f3c9cda9c3/pkg/archive/archive.go#L210 https://github.com/moby/moby/blob/c9d2b7df777b38f7239a882c27... So interesting if they implemented and tested that and get only a marginal CRC speedup. edit: someone here seems to observe a ~ doubling with unpigz vs zcat: https://unix.stackexchange.com/a/363739 https://unix.stackexchange.com/a/363739
- mxmlnkn 4y agoigzip is even faster. See also my answer that contains a quick benchmark in that linked question at the bottom.
- mxmlnkn 4y agoI recently implemented pragzip for parallel gzip decompression https://github.com/mxmlnkn/pragzip https://github.com/mxmlnkn/pragzip . I would be interested to know how your PR back then worked to parallelize the decompression.