3 ms·

I did, yes. I thought Intel stopped selling QAT cards with their newer generations of it? Personally I find that gzip is more or less strictly worse than zstd

I did, yes. I thought Intel stopped selling QAT cards with their newer generations of it? Personally I find that gzip is more or less strictly worse than zstd for any use case I can imagine, though if you can hardware offload it, then...it's free but still worse at its job. :) Though zstd can apparently be hardware offloaded if you have like, Sapphire Rapids-era QAT and the right wiring...sometimes. (I believe the demo code says it doesn't work on the streaming mode and only works on levels 1-12.) [1] [1] - https://github.com/intel/QAT-ZSTD-Plugin https://github.com/intel/QAT-ZSTD-Plugin

tecleandor 3y ago
Seems like they stop selling dedicated QAT cards, but their new E2000 DPUs (kind of a smart NIC I think your employer uses for accelerating cloud loads) have QAT instructions. But I'd say that's because those DPUs have a couple Xeons inside :) I think QAT API used to be a mess in past generations, and IIRC it's one of the factors drivers for older hardware weren't included in TrueNAS Scale. I think it was this ticket [1] where they said that it was complicated to add support for previous hardware, but can't confirm because JIRA's been acting funny since yesterday. Feels like QAT has stabilized since this Gen4 hardware and interesting things are appearing again. Not only the ZSTD plugin you linked, but I just noticed QATlib added support for lz4/lz4s past July [2]! That's interesting. They are also publishing guides for accelerating SSL operations in SSL terminators / load balancers like HAProxy [3], so between this and cloud providers using DPUs to accelerate their load, doesn't feel like they're shutting this down soon. I'd love if they make this a bit more accessible to non-developers. [1] : https://ixsystems.atlassian.net/browse/NAS-107334 [2] : https://github.com/intel/qatlib/tree/main#revision-history [3] : https://networkbuilders.intel.com/solutionslibrary/intel-quickassist-technology-intel-qat-accelerating-haproxy-performance-on-4th-gen-intel-xeon-scalable-processors-technology-guide
- rincebrain 3y ago
  Cute, the former Ubuntu ZFS developer now works for Intel on QATlib, among other things I imagine.

tecleandor 3y ago

Seems like they stop selling dedicated QAT cards, but their new E2000 DPUs (kind of a smart NIC I think your employer uses for accelerating cloud loads) have QAT instructions. But I'd say that's because those DPUs have a couple Xeons inside :) I think QAT API used to be a mess in past generations, and IIRC it's one of the factors drivers for older hardware weren't included in TrueNAS Scale. I think it was this ticket [1] where they said that it was complicated to add support for previous hardware, but can't confirm because JIRA's been acting funny since yesterday. Feels like QAT has stabilized since this Gen4 hardware and interesting things are appearing again. Not only the ZSTD plugin you linked, but I just noticed QATlib added support for lz4/lz4s past July [2]! That's interesting. They are also publishing guides for accelerating SSL operations in SSL terminators / load balancers like HAProxy [3], so between this and cloud providers using DPUs to accelerate their load, doesn't feel like they're shutting this down soon. I'd love if they make this a bit more accessible to non-developers. [1] : https://ixsystems.atlassian.net/browse/NAS-107334 [2] : https://github.com/intel/qatlib/tree/main#revision-history [3] : https://networkbuilders.intel.com/solutionslibrary/intel-quickassist-technology-intel-qat-accelerating-haproxy-performance-on-4th-gen-intel-xeon-scalable-processors-technology-guide

rincebrain 3y ago
Cute, the former Ubuntu ZFS developer now works for Intel on QATlib, among other things I imagine.