4 ms·

Good riddance; for text files, gzip is mediocre compression and fast, lzma (xz) is high compression and slow. Bzip2 is medium-high compression and _really_ slo

Good riddance; for text files, gzip is mediocre compression and fast, lzma (xz) is high compression and slow. Bzip2 is medium-high compression and _really_ slow [0]. [0] http://tukaani.org/lzma/benchmarks.html http://tukaani.org/lzma/benchmarks.html

pedrocr 13y ago
Your link shows bzip2 having a more balanced compression time than lzma that is often 3-4 times slower. So for a tarball xz sounds indeed better as you compress once and uncompress many times. For compressing HTTP or things of the sort bzip2 might be better, but maybe for those cases gzip is already best.
- jaytaylor 13y ago
  You are right, I was speaking to the compression ratios and decompression times. For cases where something is compressed once and decompressed many times I don't care much how long it takes to do the initial compression (within reason, of course).
- fiffig 13y ago
  Here is another with lz4, lzo as well as lzma1, although they're not very relevant for the tarball test. Gzip -9 is on par with bzip2 -1 while almost twice as fast. And uses practically no memory(!). http://pokecraft.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO http://pokecraft.first-world.info/wiki/Quick_Benchmark:_Gzip...
  - acdha 13y ago
    -1 is an edge-case: if you're memory constrained, bzip2 isn't a good fit. Otherwise, you pay the extremely low percentage increase in CPU time to use -9 where bz2 will massively outperform gzip on most workloads.
- Scaevolus 13y ago
  Note that LZMA decompressed at 20MB/s in that benchmark, while bzip2 managed about 6. LZMA tends to be faster and more efficient than bzip2 in all respects.
  - Buge 13y ago
    pedrocr said that LZMA decompression is faster, but that sometimes LZMA compression is slower.
sliverstorm 13y ago
I'll throw out a vote for lbzip2. It threads strongly, and doesn't cannibalize your memory like threaded xz. (You can still use gzip where every CPU cycle counts)
ketralnis 13y ago
I don't know how well lzma parallelises, but bzip2 is plenty fast for me with a multi-core version like pbzip2, as measured in wall-time on an otherwise idle system. Different considerations in different environments and all that.
- rwmj 13y ago
  Unfortunately there's no multithreaded option for the standard xz uncompression tools (xzcat, unxz). However I wrote (as a quick hack) a parallel xzcat: http://git.annexia.org/?p=pxzcat.git;a=summary http://git.annexia.org/?p=pxzcat.git;a=summary I also wrote a random access program for xz files: https://github.com/libguestfs/nbdkit https://github.com/libguestfs/nbdkit (in the plugins/xz subdirectory)
  - langarto 13y ago
    The version of the tools that I have installed (5.1.2alpha in Fedora) do support a -T option for parallel compression which I use regularly. However, it is true that that same option does not seem to work everywhere. At least, it didn't seem to work a few weeks ago in a different system (Ubuntu). The option is accepted, but does not seem to have any effect.
- vasi 13y ago
  lzma parallelizes very well! My implementation of parallel xz ( https://github.com/vasi/pixz https://github.com/vasi/pixz ) works quite well. There are others as well, including pxz and the alpha version of the standard xz. All these tools produce compressed files that conform to the xz format and can be decompressed by any xz utility.

pedrocr 13y ago

Your link shows bzip2 having a more balanced compression time than lzma that is often 3-4 times slower. So for a tarball xz sounds indeed better as you compress once and uncompress many times. For compressing HTTP or things of the sort bzip2 might be better, but maybe for those cases gzip is already best.

jaytaylor 13y ago
You are right, I was speaking to the compression ratios and decompression times. For cases where something is compressed once and decompressed many times I don't care much how long it takes to do the initial compression (within reason, of course).
fiffig 13y ago
Here is another with lz4, lzo as well as lzma1, although they're not very relevant for the tarball test. Gzip -9 is on par with bzip2 -1 while almost twice as fast. And uses practically no memory(!). http://pokecraft.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO http://pokecraft.first-world.info/wiki/Quick_Benchmark:_Gzip...
- acdha 13y ago
  -1 is an edge-case: if you're memory constrained, bzip2 isn't a good fit. Otherwise, you pay the extremely low percentage increase in CPU time to use -9 where bz2 will massively outperform gzip on most workloads.
Scaevolus 13y ago
Note that LZMA decompressed at 20MB/s in that benchmark, while bzip2 managed about 6. LZMA tends to be faster and more efficient than bzip2 in all respects.
- Buge 13y ago
  pedrocr said that LZMA decompression is faster, but that sometimes LZMA compression is slower.

sliverstorm 13y ago

I'll throw out a vote for lbzip2. It threads strongly, and doesn't cannibalize your memory like threaded xz. (You can still use gzip where every CPU cycle counts)

ketralnis 13y ago

I don't know how well lzma parallelises, but bzip2 is plenty fast for me with a multi-core version like pbzip2, as measured in wall-time on an otherwise idle system. Different considerations in different environments and all that.

rwmj 13y ago
Unfortunately there's no multithreaded option for the standard xz uncompression tools (xzcat, unxz). However I wrote (as a quick hack) a parallel xzcat: http://git.annexia.org/?p=pxzcat.git;a=summary http://git.annexia.org/?p=pxzcat.git;a=summary I also wrote a random access program for xz files: https://github.com/libguestfs/nbdkit https://github.com/libguestfs/nbdkit (in the plugins/xz subdirectory)
- langarto 13y ago
  The version of the tools that I have installed (5.1.2alpha in Fedora) do support a -T option for parallel compression which I use regularly. However, it is true that that same option does not seem to work everywhere. At least, it didn't seem to work a few weeks ago in a different system (Ubuntu). The option is accepted, but does not seem to have any effect.
vasi 13y ago
lzma parallelizes very well! My implementation of parallel xz ( https://github.com/vasi/pixz https://github.com/vasi/pixz ) works quite well. There are others as well, including pxz and the alpha version of the standard xz. All these tools produce compressed files that conform to the xz format and can be decompressed by any xz utility.