4 ms·

> I can't tell if you edited this since I read it, or I just skipped the last few paragraphs outright, my apologies. I made some minor edits within 15-30 mins

> I can't tell if you edited this since I read it, or I just skipped the last few paragraphs outright, my apologies. I made some minor edits within 15-30 mins of posting the comment, but the LZ4+zstd-1 heuristic was there from the beginning (I think) :) Since you wrote this feature, I've been wondering about something and I was hoping you could give some insights: Is there a good reason why this early abort feature wasn't implemented in zstd itself rather than ZFS? It seems like this feature could be useful even for non-filesystem use cases, right? It could make compression at higher levels substantially faster in some cases (e.g. if a significant part of your data is not compressible). Having this feature also available as an option in the zstd CLI tool could be quite useful, I think. I was thinking it could be implemented in the core zstd code and then both the CLI tool and ZFS could just enable/disable it as needed? Thank you for your great work! Edit: I just now realized, based on your Brotli comment, that since these early passes can be done (in theory) with arbitrary algorithms, perhaps it's better implemented as an additional layer on top of them rather than implementing it in each and every compression algorithm.

rincebrain 3y ago
Briefly, because I initially looked at ripping out LZ4's, but it just skips chunks in the middle of the compression algorithm, so it didn't come out easily. So I just tried using LZ4 as a first pass, and that worked well enough, with a high false positive rate, so zstd-1 was duct-taped onto it. Trying to just glue an entropy calculator on is something I want to try, but I haven't, because this worked "well enough". As far as why not do it in just zstd or LZ4 proper...well, this works very well for extremely quantized chunks, like ZFS records, but for things with the streaming mode, it becomes more exciting to generalize. Plus it's much nicer for the ZFS codebase to keep it abstracted. Also, as far as using it for, say, gzip...well, I should really open that PR from that branch I have, shouldn't I? ;)
- tecleandor 3y ago
  Ah,you were the one that gave the "Refining ZFS compression" talk, isn't it? Thanks for all your work! Works great and it's pretty interesting. I bought a super cheap QAT card with the idea of doing hardware accelerated gzip compression on ZFS (just for playing with it), but seems like the drivers are kind of a mess, and not being a developer who could debug it, maybe I should stay with the easy and fast zstd with early abort :P :) Less headaches .
  - rincebrain 3y ago
    I did, yes. I thought Intel stopped selling QAT cards with their newer generations of it? Personally I find that gzip is more or less strictly worse than zstd for any use case I can imagine, though if you can hardware offload it, then...it's free but still worse at its job. :) Though zstd can apparently be hardware offloaded if you have like, Sapphire Rapids-era QAT and the right wiring...sometimes. (I believe the demo code says it doesn't work on the streaming mode and only works on levels 1-12.) [1] [1] - https://github.com/intel/QAT-ZSTD-Plugin https://github.com/intel/QAT-ZSTD-Plugin
    - tecleandor 3y ago
      Seems like they stop selling dedicated QAT cards, but their new E2000 DPUs (kind of a smart NIC I think your employer uses for accelerating cloud loads) have QAT instructions. But I'd say that's because those DPUs have a couple Xeons inside :) I think QAT API used to be a mess in past generations, and IIRC it's one of the factors drivers for older hardware weren't included in TrueNAS Scale. I think it was this ticket [1] where they said that it was complicated to add support for previous hardware, but can't confirm because JIRA's been acting funny since yesterday. Feels like QAT has stabilized since this Gen4 hardware and interesting things are appearing again. Not only the ZSTD plugin you linked, but I just noticed QATlib added support for lz4/lz4s past July [2]! That's interesting. They are also publishing guides for accelerating SSL operations in SSL terminators / load balancers like HAProxy [3], so between this and cloud providers using DPUs to accelerate their load, doesn't feel like they're shutting this down soon. I'd love if they make this a bit more accessible to non-developers. [1] : https://ixsystems.atlassian.net/browse/NAS-107334 [2] : https://github.com/intel/qatlib/tree/main#revision-history [3] : https://networkbuilders.intel.com/solutionslibrary/intel-quickassist-technology-intel-qat-accelerating-haproxy-performance-on-4th-gen-intel-xeon-scalable-processors-technology-guide
      - rincebrain 3y ago
        Cute, the former Ubuntu ZFS developer now works for Intel on QATlib, among other things I imagine.

rincebrain 3y ago

Briefly, because I initially looked at ripping out LZ4's, but it just skips chunks in the middle of the compression algorithm, so it didn't come out easily. So I just tried using LZ4 as a first pass, and that worked well enough, with a high false positive rate, so zstd-1 was duct-taped onto it. Trying to just glue an entropy calculator on is something I want to try, but I haven't, because this worked "well enough". As far as why not do it in just zstd or LZ4 proper...well, this works very well for extremely quantized chunks, like ZFS records, but for things with the streaming mode, it becomes more exciting to generalize. Plus it's much nicer for the ZFS codebase to keep it abstracted. Also, as far as using it for, say, gzip...well, I should really open that PR from that branch I have, shouldn't I? ;)

tecleandor 3y ago
Ah,you were the one that gave the "Refining ZFS compression" talk, isn't it? Thanks for all your work! Works great and it's pretty interesting. I bought a super cheap QAT card with the idea of doing hardware accelerated gzip compression on ZFS (just for playing with it), but seems like the drivers are kind of a mess, and not being a developer who could debug it, maybe I should stay with the easy and fast zstd with early abort :P :) Less headaches .
- rincebrain 3y ago
  I did, yes. I thought Intel stopped selling QAT cards with their newer generations of it? Personally I find that gzip is more or less strictly worse than zstd for any use case I can imagine, though if you can hardware offload it, then...it's free but still worse at its job. :) Though zstd can apparently be hardware offloaded if you have like, Sapphire Rapids-era QAT and the right wiring...sometimes. (I believe the demo code says it doesn't work on the streaming mode and only works on levels 1-12.) [1] [1] - https://github.com/intel/QAT-ZSTD-Plugin https://github.com/intel/QAT-ZSTD-Plugin
  - tecleandor 3y ago
    Seems like they stop selling dedicated QAT cards, but their new E2000 DPUs (kind of a smart NIC I think your employer uses for accelerating cloud loads) have QAT instructions. But I'd say that's because those DPUs have a couple Xeons inside :) I think QAT API used to be a mess in past generations, and IIRC it's one of the factors drivers for older hardware weren't included in TrueNAS Scale. I think it was this ticket [1] where they said that it was complicated to add support for previous hardware, but can't confirm because JIRA's been acting funny since yesterday. Feels like QAT has stabilized since this Gen4 hardware and interesting things are appearing again. Not only the ZSTD plugin you linked, but I just noticed QATlib added support for lz4/lz4s past July [2]! That's interesting. They are also publishing guides for accelerating SSL operations in SSL terminators / load balancers like HAProxy [3], so between this and cloud providers using DPUs to accelerate their load, doesn't feel like they're shutting this down soon. I'd love if they make this a bit more accessible to non-developers. [1] : https://ixsystems.atlassian.net/browse/NAS-107334 [2] : https://github.com/intel/qatlib/tree/main#revision-history [3] : https://networkbuilders.intel.com/solutionslibrary/intel-quickassist-technology-intel-qat-accelerating-haproxy-performance-on-4th-gen-intel-xeon-scalable-processors-technology-guide
    - rincebrain 3y ago
      Cute, the former Ubuntu ZFS developer now works for Intel on QATlib, among other things I imagine.