4 ms·
"Modern", but uses Huffman coding? No LZ implementation aiming for high compression in the last decade has used Huffman.
by user-the-name 5y ago
"Modern", but uses Huffman coding? No LZ implementation aiming for high compression in the last decade has used Huffman.
- chriswarbo 5y ago> For our goal of a high compression LZ variant, we will want to Huffman code our symbols (zstd actually uses FSE, a variant of arithmetic coding to do this, but we will cover that in a future post).
- user-the-name 5y agoBut why? Why spend that much effort introducing a method that is basically entirely outdated at this point, and is also fairly complicated?
- hexxagone 5y agoHuffmann is rather simple and hard to beat in decompression speed.
- vardump 5y agoFor high entropy data (=somewhat random data), FSE is quite comparable to Huffman in compression speed. For low entropy (lots of high probability symbols, like zeroes), Huffman is about 2-3x faster. But on the flipside, FSE achieves markedly higher compression ratio. I think FSE is worth the speed tradeoff vs Huffman in most cases.
- hexxagone 5y ago"FSE achieves markedly higher compression ratio". I do not thin it is true, FSE/ANS achieves slightly better ratios in general. Zstd uses both Huffman (for large alphabets) and FSE (for small alphabets).
- user-the-name 5y agoArithmetic coding is much, much simpler. And decompression speed is not a limiting factor in most applications of data compression like this.
- hexxagone 5y ago"Arithmetic coding is much, much simpler." Let us agree to disagree. "And decompression speed is not a limiting factor in most applications of data compression like this". It depends on the application. Zstd and Brotli are certainly aiming at the fastest decompression speed possible.
- user-the-name 5y agoArithmetic coding can be implemented in as little as maybe ten lines of code. It is far simpler than Huffman coding.
- hexxagone 5y agoThe Huffman encoding loop is 2 lines and decoding loop is 4 lines of branchless code. Do you have an example of branchless arithmetic encoder or decoder ?
- hexxagone 5y agoYou mean like zstd and brotli ? Is there any new LZ codec not using Huffman ?
- retrac 5y agoIt was my understanding Zstd used neither Huffman or AC but something else: https://en.wikipedia.org/wiki/Asymmetric_numeral_systems https://en.wikipedia.org/wiki/Asymmetric_numeral_systems Edit: It uses a variety of entropy encodings for different data structures, Huffman is one of them. My apologies for the confusing initial comment.