4 ms·

The main reason to continue to use gzip is speed: it's an order of magnitude faster than bzip2 for compression / decompression, for a relatively small compress

by wheels 16y ago

The main reason to continue to use gzip is speed: it's an order of magnitude faster than bzip2 for compression / decompression, for a relatively small compression ratio hit. It's fast enough to us inline in web servers, SSH sessions, etc, and if you're just moving data around a single time, often the amount of time for compress / transfer / decompress is much faster than with bzip2.

alecco 16y ago
Yes, the new compression algorithms (BWT and LZ77-based) are slower and require a lot more memory than DEFLATE (*zip) in most cases. This guy did a not so rigorous analysis but it is mostly OK: http://changelog.complete.org/archives/931-how-to-think-about-compression-part-2 http://changelog.complete.org/archives/931-how-to-think-abou... A little bird told me there's a better algorithm in speed/compression rate coming soon ;)
gxti 16y ago
I've found that on many types of data, xz -2 is twice as fast and compresses better than gzip -9. Of course gzip -1 kicks the pants off xz but there's absolutely no reason to use gzip at its higher levels, nor to use bzip2 at all. And if you want fast, LZO does it better -- the lzop tool provides a typical unix compress interface. Using 50MiB from a tarball I had lying around, best of 3: prog time(s) size(%) gzip -1 1.703 38.7 gzip -9 14.670 35.1 (worse than xz -1) bzip2 -1 7.714 34.0 bzip2 -9 8.035 31.4 (worse than xz -1) xz -1 6.278 31.3 xz -9 42.445 22.7 (best compression) lzop -1 0.670 46.1 (fastest) lzop -9 18.144 38.0
- huhtenberg 16y ago
  Out of curiousity, can you run this tarball through paq (say, paq8hp12)? http://en.wikipedia.org/wiki/PAQ http://en.wikipedia.org/wiki/PAQ
  - gxti 16y ago
    It segfaulted at the end, after taking 7m28s. I don't know how big the result was because it got truncated to 35 bytes.
    - huhtenberg 16y ago
      Ugh. Can you try with a lighter compression level? There is a command line switch for that, and it should also show the memory requirements for each compression level. I'm insisting because PAQ is a really incredible collection of compressors. Far more intelligence in it than in the LZ* bunch.
- gxti 16y ago
  If anyone is still reading this thread, I made a chart overnight using a better corpus: http://partiallystapled.com/~gxti/misc/2010/06/25-shootout.png http://partiallystapled.com/~gxti/misc/2010/06/25-shootout.p... http://partiallystapled.com/~gxti/misc/2010/06/25-linux-2.6.34.ods http://partiallystapled.com/~gxti/misc/2010/06/25-linux-2.6....
kunley 16y ago
Another feature of gzip which may be your life-saver is that it keeps a record of the original size so you can know it without decompression. xz format seems to have this feature, but: $ xz --list xz: --list is not implemented yet.

by wheels 16y ago

alecco 16y ago

Yes, the new compression algorithms (BWT and LZ77-based) are slower and require a lot more memory than DEFLATE (*zip) in most cases. This guy did a not so rigorous analysis but it is mostly OK: http://changelog.complete.org/archives/931-how-to-think-about-compression-part-2 http://changelog.complete.org/archives/931-how-to-think-abou... A little bird told me there's a better algorithm in speed/compression rate coming soon ;)

gxti 16y ago

I've found that on many types of data, xz -2 is twice as fast and compresses better than gzip -9. Of course gzip -1 kicks the pants off xz but there's absolutely no reason to use gzip at its higher levels, nor to use bzip2 at all. And if you want fast, LZO does it better -- the lzop tool provides a typical unix compress interface. Using 50MiB from a tarball I had lying around, best of 3: prog time(s) size(%) gzip -1 1.703 38.7 gzip -9 14.670 35.1 (worse than xz -1) bzip2 -1 7.714 34.0 bzip2 -9 8.035 31.4 (worse than xz -1) xz -1 6.278 31.3 xz -9 42.445 22.7 (best compression) lzop -1 0.670 46.1 (fastest) lzop -9 18.144 38.0

huhtenberg 16y ago
Out of curiousity, can you run this tarball through paq (say, paq8hp12)? http://en.wikipedia.org/wiki/PAQ http://en.wikipedia.org/wiki/PAQ
- gxti 16y ago
  It segfaulted at the end, after taking 7m28s. I don't know how big the result was because it got truncated to 35 bytes.
  - huhtenberg 16y ago
    Ugh. Can you try with a lighter compression level? There is a command line switch for that, and it should also show the memory requirements for each compression level. I'm insisting because PAQ is a really incredible collection of compressors. Far more intelligence in it than in the LZ* bunch.
gxti 16y ago
If anyone is still reading this thread, I made a chart overnight using a better corpus: http://partiallystapled.com/~gxti/misc/2010/06/25-shootout.png http://partiallystapled.com/~gxti/misc/2010/06/25-shootout.p... http://partiallystapled.com/~gxti/misc/2010/06/25-linux-2.6.34.ods http://partiallystapled.com/~gxti/misc/2010/06/25-linux-2.6....

kunley 16y ago

Another feature of gzip which may be your life-saver is that it keeps a record of the original size so you can know it without decompression. xz format seems to have this feature, but: $ xz --list xz: --list is not implemented yet.