4 ms·

pigz is extremely fast and very capable, plus it's packed up and provided for almost every mainstream linux distribution, and can act as a drop in replacement f

by Twirrim 10y ago

pigz is extremely fast and very capable, plus it's packed up and provided for almost every mainstream linux distribution, and can act as a drop in replacement for gzip as it supports the same flag syntax. On the bzip2 side there is pbzip2, which is also a drop in replacement, http://compression.ca/pbzip2/ http://compression.ca/pbzip2/

treffer 10y ago
pbzip2 only does parallel compression. lbzip2 can parallelize compression and decompression. With both pbzip2 and lbzip2 I got errors every now and then while everything worked with bzip2. YMMV. Also note that bzip2 is block based (due to bwt) and thus does not compromise compression ratios like parallelized gzip implantations. At 32 cores you can saturate Gbit links (even with good compression ratios!). I hope we will see some more bzip2 love in the future due to it's perfect fit for parallelism.
- 0x0 10y ago
  Errors sound deeply concerning. What kind of errors? Is it too risky to use [pl]bzip2 for archival reasons?
  - treffer 10y ago
    Well, errors at a rate of one every few terabyte, and most of the time it was mainly decompression, and always caused program abort iirc. But those have been a few years ago, so my best advice would be: retest. You are testing your archives anyway, right? ;-)
    - paxcoder 10y ago
      >You are testing your archives anyway, right? ;-) Umm no, I handle errors which I expect to be reported to me.
sambe 10y ago
There is also pbgzip if you are into indexed access (genome data).
shabbyrobe 10y ago
pbzip2 produces a separate bzip2 stream for every compressed block, which can't be read properly by many popular decoders, usually ones that are based around the bzip2 high-level interface [0]. There's a warning about this in Python 2's documentation, which is a better offering than most [1]. Many decoders simply stop reading after decompressing the first 900k without so much as a peep that something that may astonish you has happened. [0] http://www.bzip.org/1.0.5/bzip2-manual-1.0.5.html#hl-interface [1] https://docs.python.org/2/library/bz2.html#bz2.BZ2File
thinkpad20 10y ago
There is also pixz for Xzip format. Which can be a godsend since xzip is such a slow compression. Just make sure you have the CPU to spare since it's a hog.
- voidz 10y ago
  Xz does this natively now, with the -T parameter.

by Twirrim 10y ago

treffer 10y ago

pbzip2 only does parallel compression. lbzip2 can parallelize compression and decompression. With both pbzip2 and lbzip2 I got errors every now and then while everything worked with bzip2. YMMV. Also note that bzip2 is block based (due to bwt) and thus does not compromise compression ratios like parallelized gzip implantations. At 32 cores you can saturate Gbit links (even with good compression ratios!). I hope we will see some more bzip2 love in the future due to it's perfect fit for parallelism.

0x0 10y ago
Errors sound deeply concerning. What kind of errors? Is it too risky to use [pl]bzip2 for archival reasons?
- treffer 10y ago
  Well, errors at a rate of one every few terabyte, and most of the time it was mainly decompression, and always caused program abort iirc. But those have been a few years ago, so my best advice would be: retest. You are testing your archives anyway, right? ;-)
  - paxcoder 10y ago
    >You are testing your archives anyway, right? ;-) Umm no, I handle errors which I expect to be reported to me.

sambe 10y ago

There is also pbgzip if you are into indexed access (genome data).

shabbyrobe 10y ago

pbzip2 produces a separate bzip2 stream for every compressed block, which can't be read properly by many popular decoders, usually ones that are based around the bzip2 high-level interface [0]. There's a warning about this in Python 2's documentation, which is a better offering than most [1]. Many decoders simply stop reading after decompressing the first 900k without so much as a peep that something that may astonish you has happened. [0] http://www.bzip.org/1.0.5/bzip2-manual-1.0.5.html#hl-interface [1] https://docs.python.org/2/library/bz2.html#bz2.BZ2File

thinkpad20 10y ago

There is also pixz for Xzip format. Which can be a godsend since xzip is such a slow compression. Just make sure you have the CPU to spare since it's a hog.

voidz 10y ago
Xz does this natively now, with the -T parameter.