5 ms·

LZMA2 has a better compression ratio than bzip2, according to this article http://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Markov_chain_algorithm http:/

LZMA2 has a better compression ratio than bzip2, according to this article http://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Markov_chain_algorithm http://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Mark... . Seems like it would be a good idea to switch away from bzip2/gzip to zx

wheels 16y ago
The main reason to continue to use gzip is speed: it's an order of magnitude faster than bzip2 for compression / decompression, for a relatively small compression ratio hit. It's fast enough to us inline in web servers, SSH sessions, etc, and if you're just moving data around a single time, often the amount of time for compress / transfer / decompress is much faster than with bzip2.
- alecco 16y ago
  Yes, the new compression algorithms (BWT and LZ77-based) are slower and require a lot more memory than DEFLATE (*zip) in most cases. This guy did a not so rigorous analysis but it is mostly OK: http://changelog.complete.org/archives/931-how-to-think-about-compression-part-2 http://changelog.complete.org/archives/931-how-to-think-abou... A little bird told me there's a better algorithm in speed/compression rate coming soon ;)
- gxti 16y ago
  I've found that on many types of data, xz -2 is twice as fast and compresses better than gzip -9. Of course gzip -1 kicks the pants off xz but there's absolutely no reason to use gzip at its higher levels, nor to use bzip2 at all. And if you want fast, LZO does it better -- the lzop tool provides a typical unix compress interface. Using 50MiB from a tarball I had lying around, best of 3: prog time(s) size(%) gzip -1 1.703 38.7 gzip -9 14.670 35.1 (worse than xz -1) bzip2 -1 7.714 34.0 bzip2 -9 8.035 31.4 (worse than xz -1) xz -1 6.278 31.3 xz -9 42.445 22.7 (best compression) lzop -1 0.670 46.1 (fastest) lzop -9 18.144 38.0
  - huhtenberg 16y ago
    Out of curiousity, can you run this tarball through paq (say, paq8hp12)? http://en.wikipedia.org/wiki/PAQ http://en.wikipedia.org/wiki/PAQ
    - gxti 16y ago
      It segfaulted at the end, after taking 7m28s. I don't know how big the result was because it got truncated to 35 bytes.
      - huhtenberg 16y ago
        Ugh. Can you try with a lighter compression level? There is a command line switch for that, and it should also show the memory requirements for each compression level. I'm insisting because PAQ is a really incredible collection of compressors. Far more intelligence in it than in the LZ* bunch.
  - gxti 16y ago
    If anyone is still reading this thread, I made a chart overnight using a better corpus: http://partiallystapled.com/~gxti/misc/2010/06/25-shootout.png http://partiallystapled.com/~gxti/misc/2010/06/25-shootout.p... http://partiallystapled.com/~gxti/misc/2010/06/25-linux-2.6.34.ods http://partiallystapled.com/~gxti/misc/2010/06/25-linux-2.6....
- kunley 16y ago
  Another feature of gzip which may be your life-saver is that it keeps a record of the original size so you can know it without decompression. xz format seems to have this feature, but: $ xz --list xz: --list is not implemented yet.
gaius 16y ago
LZMA is much, much slower. It is simply a matter of the rate at which you generate data and the compression ratio and the rate at which you add more storage/move the files off to tape. Can you "keep up" essentially. Right now, CPUs are fast enough that LZMA is a realistic prospect. It boils down to what the extra CPUs will cost vs the cost of the storage saved (incl. the "cost" of space in the datacentre, the administrative overhead of more hardware, etc).
- stcredzero 16y ago
  I wonder if it would be possible to have page templates delivered separately from their dynamic content? The templating of the dynamic data would happen on the browser-side machine. This way, we could use really expensive compression algorithms with high CPU cost for compression, but then cache the compression result. This could be achieved by using Javascript and fetching data as JSON, but it seems to be something that would be very useful and beneficial if it were standardized.
  - pornel 16y ago
    There is client-side XSLT already. The problem is that transformed template is usually same size or smaller than your source data. Really expensive algorithms aren't that much better to offset cost of sending template logic to the client. You could probably save some processing time by integrating compression directly into server-side templating - take advantage of the fact that some parts never change and keep them pre-compressed or at least cache some statistics about them to aid compression.
  - jhy 16y ago
    There is the SDCH spec: Shared Dictionary Compression over HTTP. Google Chrome uses it, and the Google Toolbar adds support to IE. The server supplies a pre-created dictionary (template) which gets cached, and further requests get a diff to that dictionary as a response. I've only found Google search using it on the server side, and the browser penetration is fairly low, but it seems promising. It makes requests very fast. I've also noticed that Google search serves image thumbnails as data:// URLs directly in the original response. They are going a long way to ensuring each page loads entirely in one request.
JoachimSchipper 16y ago
More importantly, xz decompresses quickly. Essentially, you get the compression ratio of bzip2 at the decompression speed of gzip (and, unfortunately, the compression speed of bzip2). EDIT: also, like gzip but unlike bzip2, you can stream data through xz (at an insignificant penalty in compression ratio).

wheels 16y ago

The main reason to continue to use gzip is speed: it's an order of magnitude faster than bzip2 for compression / decompression, for a relatively small compression ratio hit. It's fast enough to us inline in web servers, SSH sessions, etc, and if you're just moving data around a single time, often the amount of time for compress / transfer / decompress is much faster than with bzip2.

alecco 16y ago
Yes, the new compression algorithms (BWT and LZ77-based) are slower and require a lot more memory than DEFLATE (*zip) in most cases. This guy did a not so rigorous analysis but it is mostly OK: http://changelog.complete.org/archives/931-how-to-think-about-compression-part-2 http://changelog.complete.org/archives/931-how-to-think-abou... A little bird told me there's a better algorithm in speed/compression rate coming soon ;)
gxti 16y ago
I've found that on many types of data, xz -2 is twice as fast and compresses better than gzip -9. Of course gzip -1 kicks the pants off xz but there's absolutely no reason to use gzip at its higher levels, nor to use bzip2 at all. And if you want fast, LZO does it better -- the lzop tool provides a typical unix compress interface. Using 50MiB from a tarball I had lying around, best of 3: prog time(s) size(%) gzip -1 1.703 38.7 gzip -9 14.670 35.1 (worse than xz -1) bzip2 -1 7.714 34.0 bzip2 -9 8.035 31.4 (worse than xz -1) xz -1 6.278 31.3 xz -9 42.445 22.7 (best compression) lzop -1 0.670 46.1 (fastest) lzop -9 18.144 38.0
- huhtenberg 16y ago
  Out of curiousity, can you run this tarball through paq (say, paq8hp12)? http://en.wikipedia.org/wiki/PAQ http://en.wikipedia.org/wiki/PAQ
  - gxti 16y ago
    It segfaulted at the end, after taking 7m28s. I don't know how big the result was because it got truncated to 35 bytes.
    - huhtenberg 16y ago
      Ugh. Can you try with a lighter compression level? There is a command line switch for that, and it should also show the memory requirements for each compression level. I'm insisting because PAQ is a really incredible collection of compressors. Far more intelligence in it than in the LZ* bunch.
- gxti 16y ago
  If anyone is still reading this thread, I made a chart overnight using a better corpus: http://partiallystapled.com/~gxti/misc/2010/06/25-shootout.png http://partiallystapled.com/~gxti/misc/2010/06/25-shootout.p... http://partiallystapled.com/~gxti/misc/2010/06/25-linux-2.6.34.ods http://partiallystapled.com/~gxti/misc/2010/06/25-linux-2.6....
kunley 16y ago
Another feature of gzip which may be your life-saver is that it keeps a record of the original size so you can know it without decompression. xz format seems to have this feature, but: $ xz --list xz: --list is not implemented yet.

gaius 16y ago

LZMA is much, much slower. It is simply a matter of the rate at which you generate data and the compression ratio and the rate at which you add more storage/move the files off to tape. Can you "keep up" essentially. Right now, CPUs are fast enough that LZMA is a realistic prospect. It boils down to what the extra CPUs will cost vs the cost of the storage saved (incl. the "cost" of space in the datacentre, the administrative overhead of more hardware, etc).

stcredzero 16y ago
I wonder if it would be possible to have page templates delivered separately from their dynamic content? The templating of the dynamic data would happen on the browser-side machine. This way, we could use really expensive compression algorithms with high CPU cost for compression, but then cache the compression result. This could be achieved by using Javascript and fetching data as JSON, but it seems to be something that would be very useful and beneficial if it were standardized.
- pornel 16y ago
  There is client-side XSLT already. The problem is that transformed template is usually same size or smaller than your source data. Really expensive algorithms aren't that much better to offset cost of sending template logic to the client. You could probably save some processing time by integrating compression directly into server-side templating - take advantage of the fact that some parts never change and keep them pre-compressed or at least cache some statistics about them to aid compression.
- jhy 16y ago
  There is the SDCH spec: Shared Dictionary Compression over HTTP. Google Chrome uses it, and the Google Toolbar adds support to IE. The server supplies a pre-created dictionary (template) which gets cached, and further requests get a diff to that dictionary as a response. I've only found Google search using it on the server side, and the browser penetration is fairly low, but it seems promising. It makes requests very fast. I've also noticed that Google search serves image thumbnails as data:// URLs directly in the original response. They are going a long way to ensuring each page loads entirely in one request.

JoachimSchipper 16y ago

More importantly, xz decompresses quickly. Essentially, you get the compression ratio of bzip2 at the decompression speed of gzip (and, unfortunately, the compression speed of bzip2). EDIT: also, like gzip but unlike bzip2, you can stream data through xz (at an insignificant penalty in compression ratio).