4 ms·
I love the QAT libraries and I feel their abilities are overlooked. Intel also has the igzip library that does not even require QAT and it radically faster than
by jeffbee 3y ago
I love the QAT libraries and I feel their abilities are overlooked. Intel also has the igzip library that does not even require QAT and it radically faster than zlib, which is handy in older applications where gzip is unavoidable despite its obsolescence.
The major downside of course is it is quite tricky to use this stuff in practice. In the cloud, you need a bare metal instance that exposes the QAT peripheral, and they are relatively scarce. And this whole generation of hardware is only just beginning to land in public clouds. For machines you own, you will need to scrutinize Intel's somewhat ridiculous product matrix in order to acquire a Xeon that has QAT.
- VWWHFSfQ 3y ago> older applications where gzip is unavoidable despite its obsolescence Is gzip actually obsolete or are there just newer alternatives? gzip is still everywhere
- jeffbee 3y agoThe zlib implementation is not optimal in any way that I am aware of. You can get better compression with the same compute time, and you can get the same compression with less time, in every application I have seen, by using some other thing. I usually want to look at zstd, lz4, and brotli in addition to zlib. That said, the igzip re-implementation is really good. If you can use it, igzip moves gzip closer to the optimal frontier.
- rincebrain 3y agoThere's also the various zlib forks by cloudflare et al, if you want to go down that road. But yes, lz4 and zstd would be what I'd recommend people use.
- dralley 3y agoThe code quality of some of the cloudflare changes is questionable. Optimizations aren't cleanly separated by commits, some of the licensing is dicey, etc. zlib-ng tried to merge as much as they could from the cloudflare fork without getting into licensing issues.
- rincebrain 3y agoOh, sure, it's all exciting in that respect from some of those, but I was more pointing to the collection of forks as a whole as a thing to evaluate if you have to look at zlib. I would just go for lz4 and zstd, really, at this point, but if for some reason gzip is necessary or fits your use case better, ... (Though I'd really like to hear what that is.)
- jandrese 3y agoThe only place gzip wins is in compatibility. Everything supports gzip. zstd, lz4, brotli, and others aren't included by default in most places yet, although the situation is starting to improve. Plus, if you want super compatibility there is older-than-dirt pkzip/infozip.
- benatkin 3y agoit's optimal in that it's a community project and not an extremely unethical BigCorp project
- wolf550e 3y agoIt's obsolete. DEFLATE format is limited to 32KB LZ window with huffman coding. Zstd can use a much larger window (8MB recommended) and a much better entropy coder: https://github.com/Cyan4973/FiniteStateEntropy https://github.com/Cyan4973/FiniteStateEntropy
- klauspost 3y agoLast I checked QAT was limited to 64KB backreferences and independent 128KB blocks. Now you know why they are comparing 16KB payloads only.
- terrelln 3y agoYeah, that is definitely a limitation with QAT. It isn't a great fit for larger data, as it can quickly lose compression ratio due to its smaller window. However, there is a lot of compression done on data that is <64KB. E.g. compression in RocksDB. We also have some half baked ideas to combine a fast SW match finder that only looks for matches >64KB away, and supplements the matches that QAT finds.
- andrius4669 3y agoany idea how igzip's non-QAT path compares to zlib-ng[1]? [1] https://github.com/zlib-ng/zlib-ng https://github.com/zlib-ng/zlib-ng
- jeffbee 3y agoIt's fairly easy to check for yourself, but with local builds on a desktop CPU without QAT, compressing a 150MB JSON input, with igzip and minigzip (from zlib-ng) both producing a smaller output than gzip, gzip needs 1.187s, minigzip needs 0.665s, and igzip needs 0.313s.
- powturbo 3y agoYou can use TurboBench [1] to benchmark igzip, zlib-ng and others. Download TurbBench from Releases [2] Here Some Benchmarks: - https://github.com/zlib-ng/zlib-ng/issues/1486 https://github.com/zlib-ng/zlib-ng/issues/1486 - https://github.com/powturbo/TurboBench/issues/43 https://github.com/powturbo/TurboBench/issues/43 [1] https://github.com/powturbo/TurboBench https://github.com/powturbo/TurboBench [2] https://github.com/powturbo/TurboBench/releases https://github.com/powturbo/TurboBench/releases
- pizza 3y agoThis is great, I can think of several places where zlib is the bottleneck and would love to speed them up
- selectodude 3y agoIntel’s software is simply best in class (n=2).
- brirec 3y agoQAT does support SR-IOV on platforms with IOMMU support, so you can at least delegate access to QAT across multiple VMs. I don’t think any public cloud providers offer this, though.