23 ms·

pigz: A parallel implementation of gzip for multi-core machines

289 points by firloop 4y ago

josnyder 4y ago
This was great in 2012. In 2022, most use-cases should be using parallelized zstd.
- anthk 4y ago
  Gzip is everywhere.
lxe 4y ago
Protip: if you're on a massively-multicore system and need to tar/gzip a directory full of node_modules, use pigz via `tar -I pigz` or a pipe. The performance increase is incredible.
ericbarrett 4y ago
We used this to great effect at Facebook for MySQL backups in the early 2010s. The backup hosts had far more CPU than needed so it was a very nice speed-up over gzip. Eventually we switched to zstd, of course, but pigz never failed us.
- Xorlev 4y ago
  Pretty similar to that, we used pigz and netcat to bring up new MySQL read replicas in a chain at line speeds. I recall learning the technique from Tumblr's eng blog. https://engineering.tumblr.com/post/7658008285/efficiently-copying-files-to-multiple-destinations https://engineering.tumblr.com/post/7658008285/efficiently-c...
  - evanelias 4y ago
    I wrote that Tumblr eng blog post, glad to see it's still making the rounds! I later joined FB's mysql team a few years after that, although I can't quite remember if FB was still using pigz by that time. (also, hi Eric!) Separately, at Tumblr I vaguely remember examining some alternative to pigz that was consistently faster at the time (11 years ago) because pigz couldn't parallelize decompression. Can't quite remember the name of the alternative, but it had licensing restrictions which made it less attractive than pigz. Edit: the old fast alternative I was thinking of is qpress, formerly hosted at http://www.quicklz.com/ http://www.quicklz.com/ but that's no longer online. Googling it now, there are some mirrors and also looks like Percona tools used/bundled it. Not sure if they still do or if they've since switched to zstd.
    - Xorlev 4y ago
      Small world! Thanks for writing that, it was a really clever way to do it and saved me a bunch of time. :)
- antisthenes 4y ago
  Same, except we were at a small e-commerce boutique running Magento circa 2011-2013. SQL backups were simply a bash script using Pigz, running on a cron job. Simple times!
- mackman 4y ago
  Hey Eric! Hope you’re well!
jaimehrubiks 4y ago
I used this recently with -0 (no compression) to pack* billions of files into a tar file before sending them over the network. It worked amazing.
- anderskaseorg 4y ago
  Why use tar | pigz -0 when you can just use tar?
  - jaimehrubiks 4y ago
    I used tar --use-compress-program="pigz" to create the tar out of billions of files
    - richard_todd 4y ago
      But what’s confusing everyone is that tar cf - will create the tar without any external compression program needed.
      - koolba 4y ago
        Even the “f -“ option is unneeded as the default is to stream to stdout. Though it’s always a bit scary to not explicitly specify the destination in case your finger slips and the first target is itself a writeable file.
      - jaimehrubiks 4y ago
        I could definitely be wrong here, apologies for the confusion. I run many of these tasks automated, in some cases I used low compression, in others zero compression. For low compression, that command really shines, for zero compression, I would have bet I also got improvement over regular tar without compression, but again, I could be wrong here. I'll test it again
    - ac29 4y ago
      Tar is the archiver here (putting multiple files into one file), pigz with no compression isnt doing anything besides wasting CPU time.
_joel 4y ago
Use this all the time (or did when I was doing more sysadminy stuff). Useful in all sorts of backup pipelines
sitkack 4y ago
If you really want to enable all cores for compression and decompression, give pbzip2 a try. pigz isn't as parallel as pbzip2 http://compression.ca/pbzip2/ http://compression.ca/pbzip2/ *edit, as ac29 mentions below, just use zstdmt. In my quick testing it is approximately 8x faster than pbzip2 and gives better compression ratios. Wall clock time went from 41s to 3.5s for a 3.6GB tar of source, pdfs and images AND the resulting file was smaller. megs 3781 test.tar 3041 test.tar.zstd (default compression 3, 3.5s) 3170 test.tar.bz2 (default compression, 8 threads, 40s)
- ac29 4y ago
  bzip2 is very very slow though. Some types of data compress quite well with bzip, but if high compression is needed, xz is usually as good or better and natively has multithreading available. For everything else, there's zstd (also natively multithread)
  - sitkack 4y ago
    Interesting https://docs.rs/zstd/latest/zstd/stream/write/struct.Encoder.html#method.multithread https://docs.rs/zstd/latest/zstd/stream/write/struct.Encoder...
    - iruoy 4y ago
      Decompression is multithreaded by default. Compression with an argument. However it is built-in.
- walrus01 4y ago
  on the other hand, bzip2 is pretty much obsoleted now by xzip
  - booi 4y ago
    What is xzip? are you talking about xz?
omoikane 4y ago
The bit I found most interesting was actually: https://github.com/madler/pigz/blob/master/try.h https://github.com/madler/pigz/blob/master/try.h https://github.com/madler/pigz/blob/master/try.c https://github.com/madler/pigz/blob/master/try.c which implements try/catch for C99.
- dima_vm 4y ago
  But why? Most modern languages try to get rid of exceptions (Go, Kotlin, Rust).
  - Genbox 4y ago
    Kotlin does have exceptions[1] [1] https://kotlinlang.org/docs/exceptions.html#java-interoperability https://kotlinlang.org/docs/exceptions.html#java-interoperab...
    - switchbak 4y ago
      They said "try to get rid of", which to me is akin to de-emphasizing. Of course you'd have to deal with exceptions since you're running on the JVM and want to Interop with Java code, that doesn't mean it's idiomatic code.
  - jallmann 4y ago
    Golang has panic / recover / defer which are functionally similar to exceptions. It's actually a fun exercise to implement a pseudo-syntax for try/catch/finally in terms of those primitives.
    - makapuf 4y ago
      Go has exceptions but its definitely not advised to use those as an error mechanism. Recover is really a last chance effort for recovery, not a standard error catching method.
      - 4y ago
ByThyGrace 4y ago
On Linux would it Just Work™ if you aliased pigz to gzip as a drop-in replacement?
- ndsipa_pomu 4y ago
  In theory, most stuff should work as it's 99% compatible, but there might well be something that breaks. Rather than symlinking it or some such, it's better to configure the necessary tools to use the pigz command instead and then you'll at least find out what works. FWIW, I configure BackupPC to use pigz instead of gzip without any issues.
- anthk 4y ago
  Well, gzip works on extremely old .Z files too (compress).
- Twirrim 4y ago
  I've done that without issue in the past. It's argument compatible so never seemed to be a problem
walrus01 4y ago
Would not recommend using this in 2022, use zstandard or xzip instead. zstandard is faster and slightly better compression at speed selection settings that are equivalent to gzip, in addition to having the ability to compress stuff at a much greater ratio, optionally, if you allow it to take more time and cpu resources. https://gregoryszorc.com/blog/2017/03/07/better-compression-with-zstandard/ https://gregoryszorc.com/blog/2017/03/07/better-compression-...
- dspillett 4y ago
  pigz has the advantage of producing output that can be read by standard gzip processing tools (including, of course, gzip/gunzip), which are available by default on just about every OS out there so you get the faster archive creation speed without adding requirements to those who might be accessing the results later. It works because gzip streams can be tracked together as a single stream, at the start of each block is an instruction to reset the compression dictionary as if it is the start of a file/stream (which in practise it is) so you just have to concatenate the parts coming out of the parallel threads in the right order. These resets cause a small drop in overall compression rates but this is small and can be minimised by using large enough blocks.
  - walrus01 4y ago
    yes, one consideration is whether you're creating archives for your own later use, or internal use where you also have zstandard and xz handling tools. Or to send somewhere else for wider use on unknown platforms.
    - dspillett 4y ago
      Aye, pick the right tool for the target audience. If you are the target or you know everyone else who needs to read the output will have the ability to read zstd, go with that. If not consider pigz. If writing a script that others may run, have it default to gzip but use pigz if available (unless you really don't want that small % drop on compression).
  - 4y ago
rcarmo 4y ago
I chuckled at the name, since out-of-order results are a typical output of parallelization. Kudos.
- XCSme 4y ago
  I also thought the name was clever, but your comment made it even more interesting. Also, my first thought was, "is this safe to use?", I heard of gzip vulnerabilities before, but a parallel implementation sounds a lot easier to get wrong.
  - dspillett 4y ago
    Gzip streams support dictionary resets which means you can concatenate individually commuters blocks together to make a while stream. This is what pigz is doing: shooting the input into blocks, spreading the compression of these blocks over different threads so multiple cores can be used, then joining the results together in the right order. It is the very same property of the format that gzip's own --rsyncable option makes use of to stop small changes forcing a full file send when rsync (or similar) is used to transfer updated files. The idea is as simple as it is clever, one of those "why did I not think about that?" ideas that are obvious once someone else has thought of it, so adds little or no extra risk. A vulnerability that uses gzip (a "compression bomb") or can cause a gzip tool to errantly run arbitrary code, is no more likely to affect pigz than it is the standard gzip builds.
    - apetresc 4y ago
      Given that, why wouldn't this just be upstreamed into gzip? If it's a clean, simple solution that's just expanding the use of a technique that's already in the core binary?
      - cldellow 4y ago
        gzip is a pretty old, pretty core program, so I imagine it's largely in maintenance mode, and that there is a lot of friction to pushing large changes into it. At one point, pigz required the pthreads library to build. If it still does, the gzip people would need to consider if that was appropriate for them, and if not, rewrite it to be buildable without it. There are multiple implementations of zlib that are faster than the one that ships with GNU gzip, and yet they haven't been incorporated. There are also just better algorithms if compatibility with gzip isn't needed. zstd, for example, supports parallel compression, and is both faster and compresses better than gzip.
xfalcox 4y ago
One interesting trivia is that since ~2020 Docker will transparently use pigz for decompressing container image layers if it's available on the host. This was a nice speedup for us, since we use large container images and automatic scaling for incoming traffic surges.
- danuker 4y ago
  Have you optimized the low-hanging fruit in your image size? Because compression programs are as high-hanging fruit as you can get, and parallelizing them can only be done once.
- chasil 4y ago
  I think dracut also uses pigz to create the initrd when installing a new Linux kernel rpm package.
  - anistas 4y ago
    Why do you need a heavy multi-thread compressor if modern initramfs systems (like https://github.com/anatol/booster https://github.com/anatol/booster) create small image of size 2MiB and below? You won't see any improvement from parallelization on this type of data.
- mgerdts 4y ago
  pigz only parallelizes compression. Decompressing with pigz is single threaded, except perhaps a separate thread is used for crc calculation. A decade ago I implemented parallel decompresssion for pigz. This is used in Solaris kernel zone suspend and resume, which was the reason I did the work. I submitted a PR for it but madler never got around to reviewing and merging it. Since then there has been a lot of code churn that makes it a pain to apply to the current version.
  - fulafel 4y ago
    Docker is indeed looking for a "unpigz" executable to use: https://github.com/moby/moby/blob/c9d2b7df777b38f7239a882c27c763f3c9cda9c3/pkg/archive/archive.go#L210 https://github.com/moby/moby/blob/c9d2b7df777b38f7239a882c27... So interesting if they implemented and tested that and get only a marginal CRC speedup. edit: someone here seems to observe a ~ doubling with unpigz vs zcat: https://unix.stackexchange.com/a/363739 https://unix.stackexchange.com/a/363739
ananonymoususer 4y ago
I use this all the time. It's a big time saver on multi-core machines (which is pretty much every desktop made in the past 20 years). It's available in all the repos, but not included by default (at least in Ubuntu/Mint). It is most useful for compressing disk images on-the-fly while backing them up to network storage. It's usually a good idea to zero unused space first: (unprivileged commands follow) dd if=/dev/zero of=~/zeros bs=1M; sync; rm ~/zeros Compressing on the fly can be slower than your network bandwidth depending on your network speed, your processor(s) speed, and the compression level, so you typically tune the compression level (because the other two variables are not so easy to change). Example backup: (privileged commands follow) pv < /dev/sda | pigz -9 | ssh user@remote.system dd of=compressed.sda.gz bs=1M (Note that on slower systems the ssh encryption can also slow things down.) Some sharp people may notice that it's not necessarily a good idea to back up a live system this way because the filesystem is changing while the system runs. It's usually just fine on an unloaded system that uses a journaling filesystem.
- CGamesPlay 4y ago
  Alternative way of zeroing unused space without consuming all disk space: https://manpages.ubuntu.com/manpages/trusty/man8/zerofree.8.html https://manpages.ubuntu.com/manpages/trusty/man8/zerofree.8....
  - ananonymoususer 4y ago
    Thanks. If run as an unprivileged user, the dd command will not consume ALL of the disk space (so privileged processes will not be disrupted). It will consume up to the free space limit (default 5%) as described here: http://blog.serverbuddies.com/using-tune2fs-to-free-up-disk-space/ http://blog.serverbuddies.com/using-tune2fs-to-free-up-disk-... The zerofree command looks useful, but I don't know how portable it is. The dd method works across many platforms (such as AIX).
bbertelsen 4y ago
Warning for the uninitiated. Be cautious using this on a production machine. I recently caused a production system to crash because disk throughput was so high that it started delaying read/writes on a PostgreSQL server. There was panic!
jiggawatts 4y ago
Funny this comes up again so soon after I needed it! I recently did a proof-of-concept related to bioinformatics (gene assembly, etc...), and one quirk of that space is that they work with enormous text files. Think tens of gigabytes being a "normal" size. Just compressing and copying these around is a pain. One trick I discovered is that tools like pigz can be used to both accelerate the compression step and also copy to cloud storage in parallel! E.g.: pigz input.fastq -c | azcopy copy --from-to PipeBlob "https://myaccountname.blob.core.windows.net/inputs/input.fastq.gz?..." There is a similar pipeline available for s3cmd as well with the same benefit of overlapping the compression and the copy. However, if your tools support zstd, then it's more efficient to use that instead. Try the "zstd -T0" option or the "pzstd" tool for even higher throughputs but with same minor caveats. PS: In case anyone here is working on the above tools, I have a small request! What would be awesome is to automatically tune the compression ratio to match the available output bandwidth. With the '-c' output option, this is easy: just keep increasing the compression level by one notch whenever the output buffer is full, and reduce it by one level whenever the output buffer is empty. This will automatically tune the system to get the maximum total throughput given the available CPU performance and network bandwidth.
- paisleyrob 4y ago
  zstd has --adapt: --adapt[=min=#,max=#] zstd will dynamically adapt compression level to perceived I/O conditions. Compression level adaptation can be observed live by using command -v. Adaptation can be constrained between supplied min and max levels. The feature works when combined with multi-threading and --long mode. It does not work with --single-thread. It sets window size to 8 MB by default (can be changed manu‐ ally, see wlog). Due to the chaotic nature of dynamic adaptation, compressed result is not reproducible.
  - jiggawatts 4y ago
    I really should have read the documentation! That feature looks awesome, but in a quick test it could only use about 50% of the available output bandwidth. My upload speed is 50 Mbps, but zstd could only send about 25 Mbps. Similarly, on a local speed test (SSD -> SSD), using a fixed compression level was much faster than --adapt.
taf2 4y ago
Pretty sure we used or still use pigz when it's time to create a db replica...
gww 4y ago
There is another nice multi-core gzip based library called BGZF[1]. It is commonly used in bioinformatics. BGZF has the added advantage that it is block compressed with built in indexing method to permit seeking in compressed files. [1] https://github.com/samtools/htslib https://github.com/samtools/htslib
fintler 4y ago
If you ever run into the limitations of a single machine, dbz2 is also a fun little app for this sort of thing. You can run it across multiple machines and it'll automatically balance the workload across them. https://github.com/hpc/mpifileutils/blob/master/man/dbz2.1 https://github.com/hpc/mpifileutils/blob/master/man/dbz2.1
soulmachine 4y ago
I had used pigz for a few years, now I've replace it with `xz -T0`
LeoPanthera 4y ago
For maximum compression, pLzip offers lzma compression in parallel: https://www.nongnu.org/lzip/plzip.html https://www.nongnu.org/lzip/plzip.html
kristianp 4y ago
Pigz has been around for a while. Since 2007 if the copyright on this[1] page is any indication. [1] https://docs.oracle.com/cd/E88353_01/html/E37839/pigz-1.html https://docs.oracle.com/cd/E88353_01/html/E37839/pigz-1.html
powerverwirrt 4y ago
Funny, I just read about this yesterday. Time to try it on my pile of archived research data.
necovek 4y ago
Any comparative benchmarks or a write-up on the approach (other than "uses zlib and pthreads" from the README)?
- 331c8c71 4y ago
  I used it and it was noticeably faster. I didn't write down by how much.
- chasil 4y ago
  Single-threaded gzip can outperform pigz, or at least come very close, when used with GNU xargs on separate files with no dependencies. https://www.linuxjournal.com/content/parallel-shells-xargs-utilize-all-your-cpu-cores-unix-and-windows https://www.linuxjournal.com/content/parallel-shells-xargs-u... https://news.ycombinator.com/item?id=26178257 https://news.ycombinator.com/item?id=26178257
  - Xorlev 4y ago
    pigz is most useful on a single stream of data, vs. the more obviously parallel case of files without dependencies.
- mgerdts 4y ago
  Back in the day I wrote this about how it improved Solaris kernel zone suspend: https://web.archive.org/web/20160313033123/https://blogs.oracle.com/zoneszone/entry/kernel_zone_suspend_now_goes https://web.archive.org/web/20160313033123/https://blogs.ora...

289 points by firloop 4y ago

josnyder 4y ago

This was great in 2012. In 2022, most use-cases should be using parallelized zstd.

anthk 4y ago
Gzip is everywhere.

lxe 4y ago

Protip: if you're on a massively-multicore system and need to tar/gzip a directory full of node_modules, use pigz via `tar -I pigz` or a pipe. The performance increase is incredible.

ericbarrett 4y ago

We used this to great effect at Facebook for MySQL backups in the early 2010s. The backup hosts had far more CPU than needed so it was a very nice speed-up over gzip. Eventually we switched to zstd, of course, but pigz never failed us.

Xorlev 4y ago
Pretty similar to that, we used pigz and netcat to bring up new MySQL read replicas in a chain at line speeds. I recall learning the technique from Tumblr's eng blog. https://engineering.tumblr.com/post/7658008285/efficiently-copying-files-to-multiple-destinations https://engineering.tumblr.com/post/7658008285/efficiently-c...
- evanelias 4y ago
  I wrote that Tumblr eng blog post, glad to see it's still making the rounds! I later joined FB's mysql team a few years after that, although I can't quite remember if FB was still using pigz by that time. (also, hi Eric!) Separately, at Tumblr I vaguely remember examining some alternative to pigz that was consistently faster at the time (11 years ago) because pigz couldn't parallelize decompression. Can't quite remember the name of the alternative, but it had licensing restrictions which made it less attractive than pigz. Edit: the old fast alternative I was thinking of is qpress, formerly hosted at http://www.quicklz.com/ http://www.quicklz.com/ but that's no longer online. Googling it now, there are some mirrors and also looks like Percona tools used/bundled it. Not sure if they still do or if they've since switched to zstd.
  - Xorlev 4y ago
    Small world! Thanks for writing that, it was a really clever way to do it and saved me a bunch of time. :)
antisthenes 4y ago
Same, except we were at a small e-commerce boutique running Magento circa 2011-2013. SQL backups were simply a bash script using Pigz, running on a cron job. Simple times!
mackman 4y ago
Hey Eric! Hope you’re well!

jaimehrubiks 4y ago

I used this recently with -0 (no compression) to pack* billions of files into a tar file before sending them over the network. It worked amazing.

anderskaseorg 4y ago
Why use tar | pigz -0 when you can just use tar?
- jaimehrubiks 4y ago
  I used tar --use-compress-program="pigz" to create the tar out of billions of files
  - richard_todd 4y ago
    But what’s confusing everyone is that tar cf - will create the tar without any external compression program needed.
    - koolba 4y ago
      Even the “f -“ option is unneeded as the default is to stream to stdout. Though it’s always a bit scary to not explicitly specify the destination in case your finger slips and the first target is itself a writeable file.
    - jaimehrubiks 4y ago
      I could definitely be wrong here, apologies for the confusion. I run many of these tasks automated, in some cases I used low compression, in others zero compression. For low compression, that command really shines, for zero compression, I would have bet I also got improvement over regular tar without compression, but again, I could be wrong here. I'll test it again
  - ac29 4y ago
    Tar is the archiver here (putting multiple files into one file), pigz with no compression isnt doing anything besides wasting CPU time.

_joel 4y ago

Use this all the time (or did when I was doing more sysadminy stuff). Useful in all sorts of backup pipelines

sitkack 4y ago

If you really want to enable all cores for compression and decompression, give pbzip2 a try. pigz isn't as parallel as pbzip2 http://compression.ca/pbzip2/ http://compression.ca/pbzip2/ *edit, as ac29 mentions below, just use zstdmt. In my quick testing it is approximately 8x faster than pbzip2 and gives better compression ratios. Wall clock time went from 41s to 3.5s for a 3.6GB tar of source, pdfs and images AND the resulting file was smaller. megs 3781 test.tar 3041 test.tar.zstd (default compression 3, 3.5s) 3170 test.tar.bz2 (default compression, 8 threads, 40s)

ac29 4y ago
bzip2 is very very slow though. Some types of data compress quite well with bzip, but if high compression is needed, xz is usually as good or better and natively has multithreading available. For everything else, there's zstd (also natively multithread)
- sitkack 4y ago
  Interesting https://docs.rs/zstd/latest/zstd/stream/write/struct.Encoder.html#method.multithread https://docs.rs/zstd/latest/zstd/stream/write/struct.Encoder...
  - iruoy 4y ago
    Decompression is multithreaded by default. Compression with an argument. However it is built-in.
walrus01 4y ago
on the other hand, bzip2 is pretty much obsoleted now by xzip
- booi 4y ago
  What is xzip? are you talking about xz?

omoikane 4y ago

The bit I found most interesting was actually: https://github.com/madler/pigz/blob/master/try.h https://github.com/madler/pigz/blob/master/try.h https://github.com/madler/pigz/blob/master/try.c https://github.com/madler/pigz/blob/master/try.c which implements try/catch for C99.

dima_vm 4y ago
But why? Most modern languages try to get rid of exceptions (Go, Kotlin, Rust).
- Genbox 4y ago
  Kotlin does have exceptions[1] [1] https://kotlinlang.org/docs/exceptions.html#java-interoperability https://kotlinlang.org/docs/exceptions.html#java-interoperab...
  - switchbak 4y ago
    They said "try to get rid of", which to me is akin to de-emphasizing. Of course you'd have to deal with exceptions since you're running on the JVM and want to Interop with Java code, that doesn't mean it's idiomatic code.
- jallmann 4y ago
  Golang has panic / recover / defer which are functionally similar to exceptions. It's actually a fun exercise to implement a pseudo-syntax for try/catch/finally in terms of those primitives.
  - makapuf 4y ago
    Go has exceptions but its definitely not advised to use those as an error mechanism. Recover is really a last chance effort for recovery, not a standard error catching method.
    - 4y ago

ByThyGrace 4y ago

On Linux would it Just Work™ if you aliased pigz to gzip as a drop-in replacement?

ndsipa_pomu 4y ago
In theory, most stuff should work as it's 99% compatible, but there might well be something that breaks. Rather than symlinking it or some such, it's better to configure the necessary tools to use the pigz command instead and then you'll at least find out what works. FWIW, I configure BackupPC to use pigz instead of gzip without any issues.
anthk 4y ago
Well, gzip works on extremely old .Z files too (compress).
Twirrim 4y ago
I've done that without issue in the past. It's argument compatible so never seemed to be a problem

walrus01 4y ago

Would not recommend using this in 2022, use zstandard or xzip instead. zstandard is faster and slightly better compression at speed selection settings that are equivalent to gzip, in addition to having the ability to compress stuff at a much greater ratio, optionally, if you allow it to take more time and cpu resources. https://gregoryszorc.com/blog/2017/03/07/better-compression-with-zstandard/ https://gregoryszorc.com/blog/2017/03/07/better-compression-...

dspillett 4y ago
pigz has the advantage of producing output that can be read by standard gzip processing tools (including, of course, gzip/gunzip), which are available by default on just about every OS out there so you get the faster archive creation speed without adding requirements to those who might be accessing the results later. It works because gzip streams can be tracked together as a single stream, at the start of each block is an instruction to reset the compression dictionary as if it is the start of a file/stream (which in practise it is) so you just have to concatenate the parts coming out of the parallel threads in the right order. These resets cause a small drop in overall compression rates but this is small and can be minimised by using large enough blocks.
- walrus01 4y ago
  yes, one consideration is whether you're creating archives for your own later use, or internal use where you also have zstandard and xz handling tools. Or to send somewhere else for wider use on unknown platforms.
  - dspillett 4y ago
    Aye, pick the right tool for the target audience. If you are the target or you know everyone else who needs to read the output will have the ability to read zstd, go with that. If not consider pigz. If writing a script that others may run, have it default to gzip but use pigz if available (unless you really don't want that small % drop on compression).
- 4y ago

rcarmo 4y ago

I chuckled at the name, since out-of-order results are a typical output of parallelization. Kudos.

XCSme 4y ago
I also thought the name was clever, but your comment made it even more interesting. Also, my first thought was, "is this safe to use?", I heard of gzip vulnerabilities before, but a parallel implementation sounds a lot easier to get wrong.
- dspillett 4y ago
  Gzip streams support dictionary resets which means you can concatenate individually commuters blocks together to make a while stream. This is what pigz is doing: shooting the input into blocks, spreading the compression of these blocks over different threads so multiple cores can be used, then joining the results together in the right order. It is the very same property of the format that gzip's own --rsyncable option makes use of to stop small changes forcing a full file send when rsync (or similar) is used to transfer updated files. The idea is as simple as it is clever, one of those "why did I not think about that?" ideas that are obvious once someone else has thought of it, so adds little or no extra risk. A vulnerability that uses gzip (a "compression bomb") or can cause a gzip tool to errantly run arbitrary code, is no more likely to affect pigz than it is the standard gzip builds.
  - apetresc 4y ago
    Given that, why wouldn't this just be upstreamed into gzip? If it's a clean, simple solution that's just expanding the use of a technique that's already in the core binary?
    - cldellow 4y ago
      gzip is a pretty old, pretty core program, so I imagine it's largely in maintenance mode, and that there is a lot of friction to pushing large changes into it. At one point, pigz required the pthreads library to build. If it still does, the gzip people would need to consider if that was appropriate for them, and if not, rewrite it to be buildable without it. There are multiple implementations of zlib that are faster than the one that ships with GNU gzip, and yet they haven't been incorporated. There are also just better algorithms if compatibility with gzip isn't needed. zstd, for example, supports parallel compression, and is both faster and compresses better than gzip.

xfalcox 4y ago

One interesting trivia is that since ~2020 Docker will transparently use pigz for decompressing container image layers if it's available on the host. This was a nice speedup for us, since we use large container images and automatic scaling for incoming traffic surges.

danuker 4y ago
Have you optimized the low-hanging fruit in your image size? Because compression programs are as high-hanging fruit as you can get, and parallelizing them can only be done once.
chasil 4y ago
I think dracut also uses pigz to create the initrd when installing a new Linux kernel rpm package.
- anistas 4y ago
  Why do you need a heavy multi-thread compressor if modern initramfs systems (like https://github.com/anatol/booster https://github.com/anatol/booster) create small image of size 2MiB and below? You won't see any improvement from parallelization on this type of data.
mgerdts 4y ago
pigz only parallelizes compression. Decompressing with pigz is single threaded, except perhaps a separate thread is used for crc calculation. A decade ago I implemented parallel decompresssion for pigz. This is used in Solaris kernel zone suspend and resume, which was the reason I did the work. I submitted a PR for it but madler never got around to reviewing and merging it. Since then there has been a lot of code churn that makes it a pain to apply to the current version.
- fulafel 4y ago
  Docker is indeed looking for a "unpigz" executable to use: https://github.com/moby/moby/blob/c9d2b7df777b38f7239a882c27c763f3c9cda9c3/pkg/archive/archive.go#L210 https://github.com/moby/moby/blob/c9d2b7df777b38f7239a882c27... So interesting if they implemented and tested that and get only a marginal CRC speedup. edit: someone here seems to observe a ~ doubling with unpigz vs zcat: https://unix.stackexchange.com/a/363739 https://unix.stackexchange.com/a/363739

ananonymoususer 4y ago

I use this all the time. It's a big time saver on multi-core machines (which is pretty much every desktop made in the past 20 years). It's available in all the repos, but not included by default (at least in Ubuntu/Mint). It is most useful for compressing disk images on-the-fly while backing them up to network storage. It's usually a good idea to zero unused space first: (unprivileged commands follow) dd if=/dev/zero of=~/zeros bs=1M; sync; rm ~/zeros Compressing on the fly can be slower than your network bandwidth depending on your network speed, your processor(s) speed, and the compression level, so you typically tune the compression level (because the other two variables are not so easy to change). Example backup: (privileged commands follow) pv < /dev/sda | pigz -9 | ssh user@remote.system dd of=compressed.sda.gz bs=1M (Note that on slower systems the ssh encryption can also slow things down.) Some sharp people may notice that it's not necessarily a good idea to back up a live system this way because the filesystem is changing while the system runs. It's usually just fine on an unloaded system that uses a journaling filesystem.

CGamesPlay 4y ago
Alternative way of zeroing unused space without consuming all disk space: https://manpages.ubuntu.com/manpages/trusty/man8/zerofree.8.html https://manpages.ubuntu.com/manpages/trusty/man8/zerofree.8....
- ananonymoususer 4y ago
  Thanks. If run as an unprivileged user, the dd command will not consume ALL of the disk space (so privileged processes will not be disrupted). It will consume up to the free space limit (default 5%) as described here: http://blog.serverbuddies.com/using-tune2fs-to-free-up-disk-space/ http://blog.serverbuddies.com/using-tune2fs-to-free-up-disk-... The zerofree command looks useful, but I don't know how portable it is. The dd method works across many platforms (such as AIX).

bbertelsen 4y ago

Warning for the uninitiated. Be cautious using this on a production machine. I recently caused a production system to crash because disk throughput was so high that it started delaying read/writes on a PostgreSQL server. There was panic!

jiggawatts 4y ago

Funny this comes up again so soon after I needed it! I recently did a proof-of-concept related to bioinformatics (gene assembly, etc...), and one quirk of that space is that they work with enormous text files. Think tens of gigabytes being a "normal" size. Just compressing and copying these around is a pain. One trick I discovered is that tools like pigz can be used to both accelerate the compression step and also copy to cloud storage in parallel! E.g.: pigz input.fastq -c | azcopy copy --from-to PipeBlob "https://myaccountname.blob.core.windows.net/inputs/input.fastq.gz?..." There is a similar pipeline available for s3cmd as well with the same benefit of overlapping the compression and the copy. However, if your tools support zstd, then it's more efficient to use that instead. Try the "zstd -T0" option or the "pzstd" tool for even higher throughputs but with same minor caveats. PS: In case anyone here is working on the above tools, I have a small request! What would be awesome is to automatically tune the compression ratio to match the available output bandwidth. With the '-c' output option, this is easy: just keep increasing the compression level by one notch whenever the output buffer is full, and reduce it by one level whenever the output buffer is empty. This will automatically tune the system to get the maximum total throughput given the available CPU performance and network bandwidth.

paisleyrob 4y ago
zstd has --adapt: --adapt[=min=#,max=#] zstd will dynamically adapt compression level to perceived I/O conditions. Compression level adaptation can be observed live by using command -v. Adaptation can be constrained between supplied min and max levels. The feature works when combined with multi-threading and --long mode. It does not work with --single-thread. It sets window size to 8 MB by default (can be changed manu‐ ally, see wlog). Due to the chaotic nature of dynamic adaptation, compressed result is not reproducible.
- jiggawatts 4y ago
  I really should have read the documentation! That feature looks awesome, but in a quick test it could only use about 50% of the available output bandwidth. My upload speed is 50 Mbps, but zstd could only send about 25 Mbps. Similarly, on a local speed test (SSD -> SSD), using a fixed compression level was much faster than --adapt.

taf2 4y ago

Pretty sure we used or still use pigz when it's time to create a db replica...

gww 4y ago

There is another nice multi-core gzip based library called BGZF[1]. It is commonly used in bioinformatics. BGZF has the added advantage that it is block compressed with built in indexing method to permit seeking in compressed files. [1] https://github.com/samtools/htslib https://github.com/samtools/htslib

fintler 4y ago

If you ever run into the limitations of a single machine, dbz2 is also a fun little app for this sort of thing. You can run it across multiple machines and it'll automatically balance the workload across them. https://github.com/hpc/mpifileutils/blob/master/man/dbz2.1 https://github.com/hpc/mpifileutils/blob/master/man/dbz2.1

soulmachine 4y ago

I had used pigz for a few years, now I've replace it with `xz -T0`

LeoPanthera 4y ago

For maximum compression, pLzip offers lzma compression in parallel: https://www.nongnu.org/lzip/plzip.html https://www.nongnu.org/lzip/plzip.html

kristianp 4y ago

Pigz has been around for a while. Since 2007 if the copyright on this[1] page is any indication. [1] https://docs.oracle.com/cd/E88353_01/html/E37839/pigz-1.html https://docs.oracle.com/cd/E88353_01/html/E37839/pigz-1.html

powerverwirrt 4y ago

Funny, I just read about this yesterday. Time to try it on my pile of archived research data.

necovek 4y ago

Any comparative benchmarks or a write-up on the approach (other than "uses zlib and pthreads" from the README)?

331c8c71 4y ago
I used it and it was noticeably faster. I didn't write down by how much.
chasil 4y ago
Single-threaded gzip can outperform pigz, or at least come very close, when used with GNU xargs on separate files with no dependencies. https://www.linuxjournal.com/content/parallel-shells-xargs-utilize-all-your-cpu-cores-unix-and-windows https://www.linuxjournal.com/content/parallel-shells-xargs-u... https://news.ycombinator.com/item?id=26178257 https://news.ycombinator.com/item?id=26178257
- Xorlev 4y ago
  pigz is most useful on a single stream of data, vs. the more obviously parallel case of files without dependencies.
mgerdts 4y ago
Back in the day I wrote this about how it improved Solaris kernel zone suspend: https://web.archive.org/web/20160313033123/https://blogs.oracle.com/zoneszone/entry/kernel_zone_suspend_now_goes https://web.archive.org/web/20160313033123/https://blogs.ora...