4 ms·
Pigz - multicore version of gzip.
- MikeCapone 17y agoSounds good. If it works well, I'd love to see it become the new default on all *nix distros. It's about time some of these basic tools get updated for multi-core hardware.
- anotherjesse 17y agoThis is REALLY useful when you are bundling AMI images. When we changed NASA Nebula from XEN to KVM we had to tweak a few files in every image, this sped up the conversion by minutes per image.
- jey 17y agoParallel bzip2: http://compression.ca/pbzip2/ http://compression.ca/pbzip2/
- lsb 17y agoThe next parallelized compressor I'd like to see is LZMA. It's basically Lempel-Zip, like gzip, but with a much wider window (tens of megs, vs a few dozen k), and because it's dictionary-based compression it decompresses much faster than bzip.
- wmf 17y agohttp://jnovy.fedorapeople.org/pxz/ http://jnovy.fedorapeople.org/pxz/ It's definitely needed since XZ is fairly slow.
- pan69 17y ago"How is it pronounced?" Australians would pronounce it as "pig-zed". Just one of those novelties I guess... :)
- jsankey 17y agoUnless, perhaps, they have memories of watching Monkey during their childhood: http://en.wikipedia.org/wiki/Monkey_(TV_series) http://en.wikipedia.org/wiki/Monkey_(TV_series)
- abstractbill 17y agoI would guess "pigs", as in "adding cores makes pigz fly" ;)
- grannyg00se 17y agoAccording to the linked page it's pig-zee.
- jlangenauer 17y agoI'm Australian, and I'd pronounce it "pigs".
- javert 17y agoAccording to the site, it's actually pronounced "pig-zee." (I quite dislike the name, myself. And I think everyone's actually going to pronounce it "pigs".)
- Luyt 17y agoIt's in FreeBSD's ports: /usr/ports/archivers/pigz Parallel GZIP
- dryicerx 17y agoI am just curious, why not implement this within gzip instead of forking it out as a completely new project?
- jey 17y agoWhat's the benefit of that? I see downsides: have to deal with the FSF, have to work with the existing codebase, have to fight to get the changes accepted, etc. It can always be ported into gzip if that becomes desirable.
- krainboltgreene 17y agohave to fight to get the changes accepted One of the main reasons I stay the hell away from OSS unless I can fork it.
- Rexxar 17y agoThe FSF is not involved in the development of zlib. Moreover, pigz and zlib are maintained/developed by the same person.
- ulrich 17y agoThat was exactly my thought when seeing the website.
- natfriedman 17y agoWe used this in SUSE Studio to decrease build time. Worked well. There's also MPIBZIP2 which uses (IIRC) libmpi to do bzip2 compression/decompression across several machines. We messed with that too, but it wasn't worth the added complexity.
- kvs 17y agoAnyone looked at the source yet? Is it just multi-threaded or actually using parallel frameworks like ITBB or OpenMP?
- sparky 17y agoIt uses pthreads. The source is very approachable and small, if you want to dig a bit more.
- iuguy 17y agoJust out of interest, does anyone have any numbers on whether or not the problems with gzip are more i/o bound or cpu bound? I would've thought with gzip i/o would've been a bigger factor, but am happy to be proven wrong.