3 ms·

Given that LZMA is the algorithm with the highest compression ratio and lowest speed of all commonly used algorithms, this isn't particularly impressive. Why i

by jleahy 12y ago

Given that LZMA is the algorithm with the highest compression ratio and lowest speed of all commonly used algorithms, this isn't particularly impressive. Why is it better than bzip2, or gzip? Both of these are 2-3x faster than LZMA (much more so for gzip --fast) but have lower compression ratios. LZ4 would be the extreme example of an algorithm even faster than gzip (but with still lower compression ratio).

willvarfar 12y ago
The experts at http://encode.ru http://encode.ru are rather less dismissive than you. The authors of PAQ, LZ4 and everything in between hang out on that board and talk compression.
AceJohnny2 12y ago
FWIW, from the "Large Text Compression Benchmark" linked from that page: Compression Compressed size Decompresser Total size Time (ns/byte) Program Options enwik8 enwik9 size (zip) enwik9+prog Comp Decomp Mem Alg Note lzham alpha 3 x64 -m4 -d29 24,954,329 206,393,809 155,282 x 206,549,091 595 9 4800 LZ77 45 gzip 1.3.5 -9 36,445,248 322,591,995 38,801 x 322,630,796 101 17 1.6 LZ77 bzip2 1.0.2 -9 29,008,736 253,977,839 30,036 x 254,007,875 379 129 8 BWT Salient point is the decompression time (ns/byte), which is 129 for bzip2, 17 for gzip, and... 9 (!) for lzham. So on that point it blows them out of the water, while still achieving higher compression rate on this type of input. As the original webpage implies, this is perfect for stuff like video games.
- jleahy 12y ago
  That changes things, it's a shame the website doesn't explicitly say that it's faster than gzip to decompress, it just says it's faster than LZMA, which isn't saying much. Much better than lz4hc (7 ns/byte, but only compresses to 44MB): lz4hc 0.9 44,182,558 392,102,544 43,617 x 392,146,161 65 7 14 LZ77 26
- corysama 12y ago
  Is the "Mem" column saying that lzham used 4800 megabytes vs gzip using 1.6 megabytes? If that's so, it makes lzham much less attractive for console and mobile games. If that could be reduced to 8 while still being faster than gzip, then it would be very interesting. For anyone publishing compression benchmarks: Please, please include memory usage in the report! Very few reports bother to do so at the moment. Meanwhile, many compressors take the assumption that since 16 gigs of RAM is cheap these days, that means using a few extra gigs to get a speedup is totally reasonable. It may be reasonable for you, but it makes the algo useless for me.
  - usefulcat 12y ago
    The mem column is memory used for compression.
Scaevolus 12y ago
A better summary is "compression ratios within 10% of LZMA, decompression faster than gzip".
Arkanosis 12y ago
LZMA is slow to compress, indeed, but it's rather fast for decompression (given the compression ratio). While bzip2 is faster at compressing and has lower memory requirements, LZMA can beat it at decompressing speed by large margins. This is of course nowhere near LZ4.

by jleahy 12y ago

willvarfar 12y ago

The experts at http://encode.ru http://encode.ru are rather less dismissive than you. The authors of PAQ, LZ4 and everything in between hang out on that board and talk compression.

AceJohnny2 12y ago

FWIW, from the "Large Text Compression Benchmark" linked from that page: Compression Compressed size Decompresser Total size Time (ns/byte) Program Options enwik8 enwik9 size (zip) enwik9+prog Comp Decomp Mem Alg Note lzham alpha 3 x64 -m4 -d29 24,954,329 206,393,809 155,282 x 206,549,091 595 9 4800 LZ77 45 gzip 1.3.5 -9 36,445,248 322,591,995 38,801 x 322,630,796 101 17 1.6 LZ77 bzip2 1.0.2 -9 29,008,736 253,977,839 30,036 x 254,007,875 379 129 8 BWT Salient point is the decompression time (ns/byte), which is 129 for bzip2, 17 for gzip, and... 9 (!) for lzham. So on that point it blows them out of the water, while still achieving higher compression rate on this type of input. As the original webpage implies, this is perfect for stuff like video games.

jleahy 12y ago
That changes things, it's a shame the website doesn't explicitly say that it's faster than gzip to decompress, it just says it's faster than LZMA, which isn't saying much. Much better than lz4hc (7 ns/byte, but only compresses to 44MB): lz4hc 0.9 44,182,558 392,102,544 43,617 x 392,146,161 65 7 14 LZ77 26
corysama 12y ago
Is the "Mem" column saying that lzham used 4800 megabytes vs gzip using 1.6 megabytes? If that's so, it makes lzham much less attractive for console and mobile games. If that could be reduced to 8 while still being faster than gzip, then it would be very interesting. For anyone publishing compression benchmarks: Please, please include memory usage in the report! Very few reports bother to do so at the moment. Meanwhile, many compressors take the assumption that since 16 gigs of RAM is cheap these days, that means using a few extra gigs to get a speedup is totally reasonable. It may be reasonable for you, but it makes the algo useless for me.
- usefulcat 12y ago
  The mem column is memory used for compression.

Scaevolus 12y ago

A better summary is "compression ratios within 10% of LZMA, decompression faster than gzip".

Arkanosis 12y ago

LZMA is slow to compress, indeed, but it's rather fast for decompression (given the compression ratio). While bzip2 is faster at compressing and has lower memory requirements, LZMA can beat it at decompressing speed by large margins. This is of course nowhere near LZ4.