3 ms·
It's obsolete. DEFLATE format is limited to 32KB LZ window with huffman coding. Zstd can use a much larger window (8MB recommended) and a much better entropy c
by wolf550e 3y ago
It's obsolete. DEFLATE format is limited to 32KB LZ window with huffman coding.
Zstd can use a much larger window (8MB recommended) and a much better entropy coder: https://github.com/Cyan4973/FiniteStateEntropy https://github.com/Cyan4973/FiniteStateEntropy
- klauspost 3y agoLast I checked QAT was limited to 64KB backreferences and independent 128KB blocks. Now you know why they are comparing 16KB payloads only.
- terrelln 3y agoYeah, that is definitely a limitation with QAT. It isn't a great fit for larger data, as it can quickly lose compression ratio due to its smaller window. However, there is a lot of compression done on data that is <64KB. E.g. compression in RocksDB. We also have some half baked ideas to combine a fast SW match finder that only looks for matches >64KB away, and supplements the matches that QAT finds.