3 ms·
> it seems like only BZip2 parallelizes nicely Which is really unfortunate, because bzip2 is not a very good compressor otherwise. It outperforms gzip on compr
by duskwuff 5y ago
> it seems like only BZip2 parallelizes nicely
Which is really unfortunate, because bzip2 is not a very good compressor otherwise. It outperforms gzip on compression ratio, but is beat out by anything modern, and is extremely slow, even when parallelized.
A lot of modern compressors can be made parallelizable (and, as a bonus seekable!) by breaking the input/output into chunks and compressing each chunk independently. This loses a bit of compression efficiency, but surprisingly little for sufficiently large blocks.
(This isn't what pigz does, by the way. pigz keeps state between each compressed block, which prevents parallel decoding.)
> For plain old zip this is excusable
Do you mean gzip? PKZIP decompression is actually somewhat parallelizable, because each file is compressed independently of the other files in the archive -- I'm not sure anyone actually does this, though.
- jandrese 5y agoLast time I looked the information I found online said that due to the way PKZIP files are structured it isn't possible to decompress them in parallel. But checking again it seems that someone had done just that: https://github.com/jpakkane/parzip https://github.com/jpakkane/parzip
- duskwuff 5y agoThat information was incorrect. Any member in a PKZIP archive can be decompressed independently of any other files in the archive, which is what parzip does. What you can't do is decompress a single archive member in parallel.
- saagarjha 5y agoThe Xcode XIP is chunked and can be decoded in parallel, but not seekable unfortunately.