5 ms·

One interesting trivia is that since ~2020 Docker will transparently use pigz for decompressing container image layers if it's available on the host. This was a

One interesting trivia is that since ~2020 Docker will transparently use pigz for decompressing container image layers if it's available on the host. This was a nice speedup for us, since we use large container images and automatic scaling for incoming traffic surges.

danuker 4y ago
Have you optimized the low-hanging fruit in your image size? Because compression programs are as high-hanging fruit as you can get, and parallelizing them can only be done once.
chasil 4y ago
I think dracut also uses pigz to create the initrd when installing a new Linux kernel rpm package.
- anistas 4y ago
  Why do you need a heavy multi-thread compressor if modern initramfs systems (like https://github.com/anatol/booster https://github.com/anatol/booster) create small image of size 2MiB and below? You won't see any improvement from parallelization on this type of data.
mgerdts 4y ago
pigz only parallelizes compression. Decompressing with pigz is single threaded, except perhaps a separate thread is used for crc calculation. A decade ago I implemented parallel decompresssion for pigz. This is used in Solaris kernel zone suspend and resume, which was the reason I did the work. I submitted a PR for it but madler never got around to reviewing and merging it. Since then there has been a lot of code churn that makes it a pain to apply to the current version.
- fulafel 4y ago
  Docker is indeed looking for a "unpigz" executable to use: https://github.com/moby/moby/blob/c9d2b7df777b38f7239a882c27c763f3c9cda9c3/pkg/archive/archive.go#L210 https://github.com/moby/moby/blob/c9d2b7df777b38f7239a882c27... So interesting if they implemented and tested that and get only a marginal CRC speedup. edit: someone here seems to observe a ~ doubling with unpigz vs zcat: https://unix.stackexchange.com/a/363739 https://unix.stackexchange.com/a/363739
  - mxmlnkn 4y ago
    igzip is even faster. See also my answer that contains a quick benchmark in that linked question at the bottom.
- mxmlnkn 4y ago
  I recently implemented pragzip for parallel gzip decompression https://github.com/mxmlnkn/pragzip https://github.com/mxmlnkn/pragzip . I would be interested to know how your PR back then worked to parallelize the decompression.

danuker 4y ago

Have you optimized the low-hanging fruit in your image size? Because compression programs are as high-hanging fruit as you can get, and parallelizing them can only be done once.

chasil 4y ago

I think dracut also uses pigz to create the initrd when installing a new Linux kernel rpm package.

anistas 4y ago
Why do you need a heavy multi-thread compressor if modern initramfs systems (like https://github.com/anatol/booster https://github.com/anatol/booster) create small image of size 2MiB and below? You won't see any improvement from parallelization on this type of data.

mgerdts 4y ago

pigz only parallelizes compression. Decompressing with pigz is single threaded, except perhaps a separate thread is used for crc calculation. A decade ago I implemented parallel decompresssion for pigz. This is used in Solaris kernel zone suspend and resume, which was the reason I did the work. I submitted a PR for it but madler never got around to reviewing and merging it. Since then there has been a lot of code churn that makes it a pain to apply to the current version.

fulafel 4y ago
Docker is indeed looking for a "unpigz" executable to use: https://github.com/moby/moby/blob/c9d2b7df777b38f7239a882c27c763f3c9cda9c3/pkg/archive/archive.go#L210 https://github.com/moby/moby/blob/c9d2b7df777b38f7239a882c27... So interesting if they implemented and tested that and get only a marginal CRC speedup. edit: someone here seems to observe a ~ doubling with unpigz vs zcat: https://unix.stackexchange.com/a/363739 https://unix.stackexchange.com/a/363739
- mxmlnkn 4y ago
  igzip is even faster. See also my answer that contains a quick benchmark in that linked question at the bottom.
mxmlnkn 4y ago
I recently implemented pragzip for parallel gzip decompression https://github.com/mxmlnkn/pragzip https://github.com/mxmlnkn/pragzip . I would be interested to know how your PR back then worked to parallelize the decompression.