16 ms·
Pigz: Parallel gzip for modern multi-processor, multi-core machines
- eterps 4y agoI am so used to bzip2 by now, It'll be hard to switch to something else.
- Twirrim 4y agoSimilarly, for bzip2 there is pbzip2 (http://compression.great-site.net/pbzip2/?i=1 http://compression.great-site.net/pbzip2/?i=1). zstd & xz support the "-T" argument for setting thread count. If you pass "-T 0" it will attempt to detect and use a thread per physical core.
- pizza234 4y agoAFAIK (not 100% sure), multithreading support is different - parallel versions split the file in multiple segments, and compress each independently, which multithreaded functionalities apply to the same stream (no hard splitting). For this reason, there's for example pzstd, in addition to zstd.
- faeyanpiraat 4y agotangential question: compressed files look like hashes in terms of if something changes in the beginning, all the other parts are different, right?
- masklinn 4y agoUsually but it depends on the compression scheme. There’s usually a “window” of how far back they look, so they can resync after a while, but it’s unlikely and the offsets will likely have changed so you need to handle that. You can force this property by introducing synchronisation points though, gzip has an `—-rsyncable` which makes that a lot more likely, at a small compression cost. Edit: apparently zstd has also had —-rsyncable for the last 5 years.
- nine_k 4y agoFor compression efficiency, it makes sense to have one large common dictionary. For compression speed, it is easier to have a dictionary per chunk. I still hope they use the common dictionary; if so, any change in the beginning likely affects further parts if it affects the dictionary and thus the way how the later parts are compressed. Same for farther parts affecting the way earlier parts are compressed.
- MisterTea 4y agoYes, pbzip divides up the file into blocks per core. Though I think some versions (older?) of bzip are unable to handle pbzip archives. I used pbzip2 on an old octo core xeon server with a decent sas raid and was able to compress at well over 200MB/sec, closer to 300MB in some cases.
- aidenn0 4y agobzip2 is already block based though, so there is no compatibility issue in that specific case (vs pbzip2) though I think pbzip2 supports larger blocks than the original bzip2.
- gjvc 4y agoI have seen at least one case where pbzip2 created files which could not be opened by some .NET implementation of the decoder, but the same decoder could open files created by lbzip just fine. No idea why.
- jimmySixDOF 4y agoJohn Carmack just had a tweet today on this problem: >I started a tar with bzip command on a big directory, and it has been running for two days. Of course, it is only using 1.07 cores out of the 128 available. The Unix pipeline tool philosophy often isn’t aligned with parallel performance. https://twitter.com/ID_AA_Carmack/status/1656708636570271768?t=flIt5lJb6xXTWgLvoEVzAg&s=19 https://twitter.com/ID_AA_Carmack/status/1656708636570271768...
- gpderetta 4y agoBut pigz shows that the unix pipeline philosophy works just fine. (of course compressing before tarring is probably better than compressing the tarred file, but that should be pipelinable as well)
- viraptor 4y ago> compressing before tarring is probably better Not if the files are similar. If you're compressing the files separately you'll start with a clean state rather than reusing previous fragments. Compressing a BMP after a TXT may not be beneficial, but compressing 3 tar'ed TXTs is definitely better than doing them separately.
- zweifuss 4y agoThat is true and seems easily half the total size with small and similar files, but it also means you have to unpack the whole archive when you need the last file in a tarball. AFAIK the gzip command still cannot compress directory information and therefore needs tar in front of it if you want to retain a folder structure.
- cout 4y agoFor many small files, compressing first will compress worse, because each file has its own dictionary; compressing last means you can take advantage of similarities in files to improve compression ratio. Compressing first can also be slower if the average file size is smaller than the block size, because the main thread cannot queue new jobs as fast as cores complete them (this happens e.g. with 7zip at fast compression settings with solid archive turned off). Tarring then compressing means small files can be aggregated into a single block, giving both good speed and compression ratio.
- osivertsson 4y agoUseful with Docker, see https://github.com/moby/moby/pull/35697 https://github.com/moby/moby/pull/35697 I’ve integrated pigz into different build and CI pipelines a few times. Don’t expect wonders since some steps still need to run serially, but a few seconds here and there might still add up to a few minutes on a large build.
- mikepurvis 4y agoI built a custom dpkg with parallel xz for speeding up the compression of large omni style deb packages. Totally worth it.
- mikepurvis 4y agoTurns out it was longer ago than I thought— way back in the Ubuntu 14.04 and 16.04 days: https://launchpad.net/~mikepurvis/+archive/ubuntu/dpkg https://launchpad.net/~mikepurvis/+archive/ubuntu/dpkg
- macNchz 4y agoAm I reading correctly that Docker just automatically uses pigz if it’s in the system path? I’ve used both for years and had no idea. I’m definitely going to make sure it’s installed in CI pipelines going forward, I know of some bloated image builds it will definitely help with!
- cpuguy83 4y agoCorrect, if it detects in unpigz it will use it. It will not compress layers with it, though.
- stabbles 4y agoAlso works with zlib-ng
- gpderetta 4y agoThe issue with pigz is that uncompressing doesn't really parallelize beyond a three stage read/uncompress/write pipeline. This is of course more of a problem of the gz format than pigz although last time I looked hacks are possible to parallelize decompression.
- mgerdts 4y agoI implemented parallel decompression a while back. It is in Solaris 11.3 and later. https://github.com/oracle/solaris-userland/blob/master/components/pigz/patches/202_index.patch https://github.com/oracle/solaris-userland/blob/master/compo... Shortly after submitting a PR the code went through major surgery, and my patch then needed a similar amount of surgery. Oracle then whacked most of the Solaris org, and I don’t think this ever got updated to work with the current pigz.
- gpderetta 4y agoNice! You should be able to do it without an index by periodically restarting the dictionary on compression and then looking for something resembling the dictionary, right?
- mgerdts 4y agoYeah, probably so at the cost of compatibility. As implemented, the .gz file can be used with `gzip -d`.
- gcflymoto 4y agoWould you mind creating a fork of pigz in GitHub and add this patch? I would be interested in testing it out!
- mgerdts 4y agoYou can grab the version from the solaris userland repo I linked and use it without me completing a homework assignment. Just grab the pigz-2.3.4 source then apply the patches from [1] in the proper order. Maybe some of them aren't needed for non-Solaris. 1. https://github.com/oracle/solaris-userland/tree/master/components/pigz/patches https://github.com/oracle/solaris-userland/tree/master/compo... I thought I had opened a PR for that a long while ago, but it doesn't show up on github these days. In any case, I did ask Mark Adler to review it. It was never a priority, then the code changed in ways that I don't really want to deal with. While looking through the PRs, I noticed a PR for Blocked GZip Format (BGZF) [2]. That's very interesting, and perhaps suggests that bgzip is a tool you would be interested in. 2. https://github.com/madler/pigz/pull/19 https://github.com/madler/pigz/pull/19
- sllabres 4y agoHardware accelerated [1] "GZIP Acceleration with AIX on Power Systems " pigz [1] https://community.ibm.com/community/user/power/blogs/brian-veale1/2022/03/14/gzip-acceleration-with-aix-on-power-systems https://community.ibm.com/community/user/power/blogs/brian-v...
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- fleetside72 4y agoI was mostly interested in the name and the pronunciation section kind of ruined it for me
- kps 4y agoIf it's to be pig-zee (pixie) for the Americans, it can be pig-zed (pig's head) for the rest of us.
- lkuty 4y agoThat's funny. Exactly, in french we pronounce z like zed and so that allows us to keep the funny part. Even funnier than "pigs". Thanks
- lkuty 4y agoI will pronounce it like "pigs" anyway. More fun.
- nammi 4y agoThat's really confusing since `pixz` exists and its "pixie" pronunciation actually works https://github.com/vasi/pixz https://github.com/vasi/pixz
- adamgordonbell 4y agoI heard of pigz in the discussions following my interview of Yann Collet, creator of LZ4 and zstd. If you'll excuse the plug, here is the LZ4 story: Yann was bored and working as a project manager. So he started working on a game for his old HP 48 graphing calculator. Eventually, this hobby led him to revolutionize the field of data compression, releasing LZ4, ZStandard, and Finite State Entropy coders. His code ended up everywhere: in games, databases, file systems, and the Linux Kernel because Yann built the world's fastest compression algorithms. And he got started just making a fun game for a graphing calculator he'd had since high school. https://corecursive.com/data-compression-yann-collet/ https://corecursive.com/data-compression-yann-collet/
- lionkor 4y agoI loved this episode, it was very engaging to the very end, I wish there were more episodes! (I already listened to all of them so far) Thank you for doing this podcast!
- adamgordonbell 4y agoOne wild thing is how much performance wins were available compared to ZLib. Pigz is parrellel, but what if you just had a better way to compress and decompress than DEFLATE? When zstd came out – and Brotli before it to a certain extent – they were 3x faster than ZLib with a slightly higher compression ratio. You'd think that such performance jumps in something as well explored as data compression would be hard to come by. We weren't that close to the efficiency frontier.
- henrydark 4y agoEven though the opposite is recited more frequently, in some cases good can be the enemy of perfect
- dist-epoch 4y ago> as well explored as data compression What's well explored is compression rate, where indeed it's difficult to improve, and true innovations, like arithmetic coding, are rare. Compressing speed on the other hand it's not very interesting to academics, it's more of an engineering problem. And there is plenty of work to do here, starting with stuff as simple as multi-threading and SIMD. ZLib and ZStandard are probably in the same complexity class, but with different constant factors, which academics don't care about but which have massive practical consequences.
- vr46 4y agoBecause I can can never remember to use pigz I have to have this in my dotfiles: function ccm() { tar -cf - $1 | pigz > $1.tar.gz }
- mmh0000 4y agoGood thing you never use file names with spaces ;)
- mmh0000 4y agoOkay, I feel that my first response was too snarky. I'm sorry. In its place, I'll say: You wouldn't go outside without pants. You shouldn't use a variable without quotes. Put pants on all variables. Also, you shouldn't use () with 'function' it "works" in Bash but it's not standard: ccm() { tar c "${1}" | pigz > "${1}.tar.gz" } You could further improve it with a loop to accept multiple files: ccm() { for i in $@; do tar c "${i}" | pigz > "${i}.tar.gz" done } Now you can run it like: `ccm file1 file2 file3` See: https://mywiki.wooledge.org/Quotes https://mywiki.wooledge.org/Quotes See: https://mywiki.wooledge.org/BashGuide/CompoundCommands#Functions https://mywiki.wooledge.org/BashGuide/CompoundCommands#Funct...
- joaonmatos 4y agoBest of luck with the implementation, but I do hope the authors realise they should avoid naming their software tools like an old-school pornographic film. I thought we had learned that from the GIMP[1]. [1] https://www.theregister.com/2019/08/28/gimp_open_source_image_editor_forked_to_fix_problematic_name/ https://www.theregister.com/2019/08/28/gimp_open_source_imag...
- loeg 4y agoThere is nothing pornographic about “pigz.”
- joaonmatos 4y agoPigs elicits images of heavy exchange of bodily fluids between the intervenients
- ComodoHacker 4y ago> exploits multiple processors and multiple cores to the hilt when compressing data As s side note, this isn't always desirable for this class of coders. In some scenarios (like web server) you might want to favor throughput over response time.
- ElectricalUnion 4y ago`zstd --adapt` is pretty cool as it detects how much output buffer it has and changes compression effort on the fly to try to achieve maximum throughput.
- loeg 4y agoI think GP is talking about the case where you have more than one client and don’t want to throw all the server’s threads at serving just one.
- jdblair 4y agoI'm a big fan of pigz. I use it in my home-grown backup script for my Linux laptop. It can compress the incremental tar output from my filesystem snapshot fast enough to saturate the I/O to my external USB3 hard drive. This is a low bar, but single-threaded gzip (or bzip2) could not do it!
- daneel_w 4y agoGive xz and zstd a go. You'll love them.
- ljosifov 4y agoI update/upgrade/switch over to zstd (from older compressors) wherever I'm updating or revamping any of my data pipelines. Looks like a win^3 for me: 1) It's probably either in the top-X or #1 in any of the usual compression metrics size/speed/convenience/ease etc. 2) Can do --rsyncable and create rsync friendly files at tiny size cost. 3) In the rare occasion I need there's $ zstd -c file1 >file.zst; zstd -c file2 >>file.zst, then $ zstd -dc file.zst will produce out $ cat file{1,2}
- gildas 4y agoSimilarly, for zipping files in JS, I have coded the possibility to compress zip files on several cores in zip.js [1]. The approach is simpler as it consists of compressing the entries in parallel. It still offers a significant performance gain though when compressing multiple files in a zip file, which is often the nominal case. [1] https://github.com/gildas-lormeau/zip.js https://github.com/gildas-lormeau/zip.js
- mgerdts 4y agoIf you are interested in optimizing parallel decompression and you happen to have a suitable NVIDIA GPU, GDeflate [1] is interesting. The target market for this is PC games using DirectStorage to quickly load game assets. The graph in [1] shows DirectStorage maxing out the throughput of a PCIe Gen 3 drive at about 3 GiB/s when compression is not used. When GPU GDeflate is used, the effective rate hits 12 GiB/s. If you have suitable hardware running Windows, you can try this out for yourself using Microsoft's DirectStorage GPU decompression benchmark [2]. A reference implementation of a single threaded compressor and multi (CPU) threaded decompressor can be found at [3]. It is Apache-2 licensed. 1. https://developer.nvidia.com/blog/accelerating-load-times-for-directx-games-and-apps-with-gdeflate-for-directstorage/ https://developer.nvidia.com/blog/accelerating-load-times-fo... 2. https://github.com/microsoft/DirectStorage/tree/main/Samples/GpuDecompressionBenchmark https://github.com/microsoft/DirectStorage/tree/main/Samples... 3. https://github.com/microsoft/DirectStorage/blob/main/GDeflate/GDeflate https://github.com/microsoft/DirectStorage/blob/main/GDeflat... Disclaimer: I work for NVIDIA, have nothing to do with this, and am not speaking for NVIDIA. Edit: oops, lost the last sentence in the first paragraph during an edit.
- gumballindie 4y agoPosted a question (now deleted) asking if it could be done on the gpu not noticing you already posted this. Thanks for sharing.
- Retr0id 4y ago> the effective rate hits 12 GiB/s I assume this is for decompressing multiple independent deflate streams in parallel? What's the throughput if you only have a single stream? I realise this is the unhappy-case for GPU acceleration, hence my question! (I've been thinking about some approaches to parallelize decompression of a single stream, it's not easy)
- mgerdts 4y ago> I've been thinking about some approaches to parallelize decompression of a single stream, it's not easy You saw this, right? https://news.ycombinator.com/item?id=35915285 https://news.ycombinator.com/item?id=35915285
- ctur 4y agoUnless the recipient of whatever you are compressing absolutely requires gzip, you should not use gzip or pigz. Instead you should use zstd as it compresses faster, decompresses faster, and yields smaller files. It also supports parallelism (via “-T”) which supplants the pigz use case. There literally are no trade-offs; it is better in every objective way. In 2023, friends don’t let friends use gzip.
- polski-g 4y ago`tar czf` is a lot easier to remember than `tar -I zstd cf`
- aendruk 4y agotar --create --zstd --file
- 5e92cb50239222b 4y agoGNU tar can autodetect the compression algorithm, both for compression and decompression. $ tar -caf dst.tar.zst /src $ tar -xaf src.tar.zst (it's fine to omit -a for decompression)
- piperswe 4y agoI never remember either, so I might as well look up the latter rather than the former.
- capableweb 4y ago> There literally are no trade-offs; it is better in every objective way. There literally are trade-offs, you started your comment describing one of them. If you want as wide out-of-the-box support as possible, you'd go with gzip. The Compression Streams browser API only supports gzip (+ deflate) so if you wanna compress something natively in the browser without 3rd party libraries (or slow JS implementation), gzip seems to be the only option.
- daneel_w 4y ago
- vrglvrglvrgl 4y ago[dead]
- draxil 4y ago> It is not pronounced like the plural of pig. I've got news for you buddy :)
- tysam_and 4y agoI see someone else read the Carmack post complaining about single threaded compression performance on Unix. Hopefully my tweet response was the one to tip you off! ;P Though in all likelihood I'm quite sure a number of people commented pointing at pigz. Hats off to all who write extraordinarily performant multithreaded versions of originally-slow-at-scale UNIX system tools.
- wild_pointer 4y agoHow does it compare perfomance-wise to Intel ISA-L? https://github.com/intel/isa-l https://github.com/intel/isa-l
- 2809 4y agoStill blows my mind people still use gzip. 20 years ago I was expecting by this point in time for there to be lots of effort put into increasing compression and working towards getting that fast, instead its been a push for speed. It makes sense with how the internet has changed. These days gzip isn't even in the top 100 as far as compression goes, hell even something like RAR or 7zip are far back compared to the best. Take something like enwik8 (100megs), gzip will get that down to 36megs, with LZMA down to ~24-25. The top of the line stuff will get it down to the ~15meg range. Thats a huge difference.
- stjohnswarts 4y agoIt shouldn't, it's still plenty fast for 95% of the stuff out there that you wanna make a zipped archive of. It's pretty much installed everywhere too
- cmckn 4y agoContainerd will utilize unpigz if it’s on your PATH, thank me later: https://github.com/containerd/containerd/blob/main/archive/compression/compression.go#L299-L323 https://github.com/containerd/containerd/blob/main/archive/c...
- tgtweak 4y agoI always install aria2c and set package manager + wget to use it for any system file downloads... basically it will open X connections to download files based on the file size and in the process give a pretty notable speed up on those slow single-connection package repos or download URLs. For reference it can cut 2-3 minutes off of an ubuntu dist-upgrade and even more if you're on a fast-but-far connection.
- 2-718-281-828 4y agoI'm starting to get sick of those cartoon project names. Not sure what the alternative would be but it's increasingly rubbing me the wrong way.
- tgtweak 4y agopigz ... parallel implementation of gzip - that's not even a stretch of the meaning. Also - you'll never forget it.
- 2-718-281-828 4y ago"gzip-parallel"?
- tgtweak 4y agoI remember moving a HUGE mysql table (>500GB) with a pipe chain of mysqldump > pigz > scp (compression disabled) > pigz > mysql If you've ever screwed around with mysqldump -> tar -> scp -> untar -> mysql<, you'll appreciate the speedup on this, in cases where you're setting up a slave and want to have the freshest possible data before kicking off binlog replication - this is the best.
- deleted 4y ago[deleted]
- kunley 4y agoIt's a very useful piece of software for over a decade. I am only disappointed with this one: "It is not pronounced like the plural of pig." Me and my colleagues always pronounced it like pigs, die Schweine - and it was so much fun!
- CreRecombinase 4y agoDoes pigz offer any advantages over bgzip?
- PaulHoule 4y agoMy understanding is that this all works because you can concat two gzip files and the outcome is the same as concatenating the original files $ gzip -c a > a.gz $ gzip -c b > b.gz $ cat a b > c1 $ cat a.gz b.gz> c.gz $ gzip -dc c.gz > c2 $ cmp c1 c2 [no output, files match]
- alfalfasprout 4y agoWhile pigz is great as a general replacement for gzip, for most purposes nowadays either LZ4 or zstd are better choices for fast compression+decompression.
- Piezoid 4y agoOne approach is to perform the bitstream parsing and Huffman decoding concurrently, while carrying out the LZ77 decoding sequentially. This method does not rely on a specialized encoder, such as pigz, that isolates LZ references into chunks.
- michaelcampbell 4y ago> I'm glad you asked. It is pronounced “pig-zee”. It is not pronounced like the plural of pig. Then don't spell it like that. And I'll continue to say GIF with a hard G also.