I imagine for some reason their encryption or some other processing step is not performed by their hardware (you can get NICs with crypto offload) or they've fo

I imagine for some reason their encryption or some other processing step is not performed by their hardware (you can get NICs with crypto offload) or they've found it cheaper or more flexible to go with doing it on the CPU. But for other, this is what is done. Which is probably why their use case here isn't optimized all that well. There have been ongoing efforts on and off to make DMA directly to/from CPU cache work well. And it can be made to work with some hardware, but I don't know if it's performed to potential. In theory his workload requires no DRAM accesses for the data if the working set could fit in cache. DMA from disk to dirty cache line, CPU loads from there encrypts, stores it in place or to new dirty line, NIC loads from that dirty data, and the dirty cache lines are invalidated or overwritten in a ring buffer with new data to avoid the mandatory writeback to DRAM. Some of the new CXL protocols coming down the pipeline might improve this situation, but they also might not.