3 ms·

It's very annoying that of all of the commonly used compression formats in use today it seems like only BZip2 parallelizes nicely. Some will use all of the cor

It's very annoying that of all of the commonly used compression formats in use today it seems like only BZip2 parallelizes nicely. Some will use all of the cores on compression but only one for decompression and others do all of the math on a single core. For plain old zip this is excusable since it comes from the 80s, but for modern compression utilities they really should do better.

paulmd 5y ago
Easy to say, hard to do. Particularly since most threading optimizations will trade some bandwidth - and people generally prefer smaller file sizes as their first-order preference.
- jandrese 5y ago
  The files are typically within a few bytes, there's really no practical difference in size.
andrelaszlo 5y ago
Switching from gzip->pigz really sped things up in one of our CI steps recently. It seemed like the output files were identical as well. What are the drawbacks?
duskwuff 5y ago
> it seems like only BZip2 parallelizes nicely Which is really unfortunate, because bzip2 is not a very good compressor otherwise. It outperforms gzip on compression ratio, but is beat out by anything modern, and is extremely slow, even when parallelized. A lot of modern compressors can be made parallelizable (and, as a bonus seekable!) by breaking the input/output into chunks and compressing each chunk independently. This loses a bit of compression efficiency, but surprisingly little for sufficiently large blocks. (This isn't what pigz does, by the way. pigz keeps state between each compressed block, which prevents parallel decoding.) > For plain old zip this is excusable Do you mean gzip? PKZIP decompression is actually somewhat parallelizable, because each file is compressed independently of the other files in the archive -- I'm not sure anyone actually does this, though.
- jandrese 5y ago
  Last time I looked the information I found online said that due to the way PKZIP files are structured it isn't possible to decompress them in parallel. But checking again it seems that someone had done just that: https://github.com/jpakkane/parzip https://github.com/jpakkane/parzip
  - duskwuff 5y ago
    That information was incorrect. Any member in a PKZIP archive can be decompressed independently of any other files in the archive, which is what parzip does. What you can't do is decompress a single archive member in parallel.
- saagarjha 5y ago
  The Xcode XIP is chunked and can be decoded in parallel, but not seekable unfortunately.
rbanffy 5y ago
> it seems like only BZip2 parallelizes nicely. That's odd. Zip and Rar, at least, have indexes that point to individual files in the archive and it should be trivial to decompress files in a thread pool with as many cores you want. That's not as easy when you have a tar.bz or tar.gz2 file because you only see files as the .tar file is extracted.
mc10 5y ago
I believe zstd does parallel compression/decompression and has some of the best efficiency ratios.[1] I’ve been replacing gzip with zstd as a general purpose compression format. [1]: Benchmarks available at http://facebook.github.io/zstd/#benchmarks http://facebook.github.io/zstd/#benchmarks
- out_of_protocol 5y ago
  So, zstd is gzip replacement? lower compression ratio than bzip2 and much much faster. Also better in both compression and speed than gzip
  - bradknowles 5y ago
    With the right options, zstd can compress quite a bit better than bzip2 and gzip. Albeit probably not as fast as gzip, because squeezing out that extra compression space is a trade for speed.

paulmd 5y ago

Easy to say, hard to do. Particularly since most threading optimizations will trade some bandwidth - and people generally prefer smaller file sizes as their first-order preference.

jandrese 5y ago
The files are typically within a few bytes, there's really no practical difference in size.

andrelaszlo 5y ago

Switching from gzip->pigz really sped things up in one of our CI steps recently. It seemed like the output files were identical as well. What are the drawbacks?

duskwuff 5y ago

> it seems like only BZip2 parallelizes nicely Which is really unfortunate, because bzip2 is not a very good compressor otherwise. It outperforms gzip on compression ratio, but is beat out by anything modern, and is extremely slow, even when parallelized. A lot of modern compressors can be made parallelizable (and, as a bonus seekable!) by breaking the input/output into chunks and compressing each chunk independently. This loses a bit of compression efficiency, but surprisingly little for sufficiently large blocks. (This isn't what pigz does, by the way. pigz keeps state between each compressed block, which prevents parallel decoding.) > For plain old zip this is excusable Do you mean gzip? PKZIP decompression is actually somewhat parallelizable, because each file is compressed independently of the other files in the archive -- I'm not sure anyone actually does this, though.

jandrese 5y ago
Last time I looked the information I found online said that due to the way PKZIP files are structured it isn't possible to decompress them in parallel. But checking again it seems that someone had done just that: https://github.com/jpakkane/parzip https://github.com/jpakkane/parzip
- duskwuff 5y ago
  That information was incorrect. Any member in a PKZIP archive can be decompressed independently of any other files in the archive, which is what parzip does. What you can't do is decompress a single archive member in parallel.
saagarjha 5y ago
The Xcode XIP is chunked and can be decoded in parallel, but not seekable unfortunately.

rbanffy 5y ago

> it seems like only BZip2 parallelizes nicely. That's odd. Zip and Rar, at least, have indexes that point to individual files in the archive and it should be trivial to decompress files in a thread pool with as many cores you want. That's not as easy when you have a tar.bz or tar.gz2 file because you only see files as the .tar file is extracted.

mc10 5y ago

I believe zstd does parallel compression/decompression and has some of the best efficiency ratios.[1] I’ve been replacing gzip with zstd as a general purpose compression format. [1]: Benchmarks available at http://facebook.github.io/zstd/#benchmarks http://facebook.github.io/zstd/#benchmarks

out_of_protocol 5y ago
So, zstd is gzip replacement? lower compression ratio than bzip2 and much much faster. Also better in both compression and speed than gzip
- bradknowles 5y ago
  With the right options, zstd can compress quite a bit better than bzip2 and gzip. Albeit probably not as fast as gzip, because squeezing out that extra compression space is a trade for speed.