3 ms·

I wonder how it compares to Intel's custom zlib implementation, and to a fast zlib compressor by IBM researchers. http://www.intel.com/content/dam/www/public/u

by pdknsk 12y ago

I wonder how it compares to Intel's custom zlib implementation, and to a fast zlib compressor by IBM researchers. http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/zlib-compression-whitepaper-copy.pdf http://www.intel.com/content/dam/www/public/us/en/documents/... http://dx.doi.org/10.1109/DCC.2014.66 http://dx.doi.org/10.1109/DCC.2014.66

twotwotwo 12y ago
Zstandard beats the Intel zlib tweak (as does Google's Gipfeli, mentioned in another comment) -- Intel's zlib -6 could also be described as "faster than stock zlib with only a little compression loss", but (from the chart in the Intel paper) it's "only" 1.8x faster when Zstandard is closer to 10x. The IBM implementation's abstract mentions "a factor of 2.6x or higher" so similar deal, unless the "or higher" language hides a 10x gain. Also, abstract says they did it by taking LZ4's matcher + Zlib's entropy encoder, so they can't be expected to outperform the LZ4 author's new work. :) Of course, those are still some great backwards-compatible gains. Could be that on modern processors, Huffman coding just isn't the best option to make your compressor scream. Gipfeli uses a simple non-Huffman entropy code, and Collet (author of Zstandard) has been working on a state-machine-based coding approach for a while. deflate is over 20 years old, so it would make sense for it to be designed for different realities. Speaking of chipmakers and zlib, though, Intel sells "QuickAssist" dedicated compression hardware, and AMD promises a compression accelerator in their server ARM SoC. As with AES, performance in software might come to matter less if systems start doing it in hardware. I don't know the silicon-area requirements for a compression accelerator, though (whereas AES's design was hardware-optimized), and I haven't seen any suggestion that compression accelerators are on the roadmap to be in any consumer chips.

by pdknsk 12y ago

twotwotwo 12y ago

Zstandard beats the Intel zlib tweak (as does Google's Gipfeli, mentioned in another comment) -- Intel's zlib -6 could also be described as "faster than stock zlib with only a little compression loss", but (from the chart in the Intel paper) it's "only" 1.8x faster when Zstandard is closer to 10x. The IBM implementation's abstract mentions "a factor of 2.6x or higher" so similar deal, unless the "or higher" language hides a 10x gain. Also, abstract says they did it by taking LZ4's matcher + Zlib's entropy encoder, so they can't be expected to outperform the LZ4 author's new work. :) Of course, those are still some great backwards-compatible gains. Could be that on modern processors, Huffman coding just isn't the best option to make your compressor scream. Gipfeli uses a simple non-Huffman entropy code, and Collet (author of Zstandard) has been working on a state-machine-based coding approach for a while. deflate is over 20 years old, so it would make sense for it to be designed for different realities. Speaking of chipmakers and zlib, though, Intel sells "QuickAssist" dedicated compression hardware, and AMD promises a compression accelerator in their server ARM SoC. As with AES, performance in software might come to matter less if systems start doing it in hardware. I don't know the silicon-area requirements for a compression accelerator, though (whereas AES's design was hardware-optimized), and I haven't seen any suggestion that compression accelerators are on the roadmap to be in any consumer chips.