4 ms·
> Things like inline-kTLS (where the NIC does the TLS encryption) reduce the memory bandwidth requirements by roughly half If storage is accessed via PCI-e, co
by magicalhippo 5y ago
> Things like inline-kTLS (where the NIC does the TLS encryption) reduce the memory bandwidth requirements by roughly half
If storage is accessed via PCI-e, couldn't one just pipe it straight from the storage to the NIC, using p2pdma[1][2] or similar? Seems you could skip all of the main DRAM round-trip this way.
Or aren't things that simple?
[1]: https://www.kernel.org/doc/html/latest/driver-api/pci/p2pdma.html https://www.kernel.org/doc/html/latest/driver-api/pci/p2pdma...
[2]: https://youtu.be/LDOlqgUZtHE?t=612 https://youtu.be/LDOlqgUZtHE?t=612
- throwawaylinux 5y agoI imagine for some reason their encryption or some other processing step is not performed by their hardware (you can get NICs with crypto offload) or they've found it cheaper or more flexible to go with doing it on the CPU. But for other, this is what is done. Which is probably why their use case here isn't optimized all that well. There have been ongoing efforts on and off to make DMA directly to/from CPU cache work well. And it can be made to work with some hardware, but I don't know if it's performed to potential. In theory his workload requires no DRAM accesses for the data if the working set could fit in cache. DMA from disk to dirty cache line, CPU loads from there encrypts, stores it in place or to new dirty line, NIC loads from that dirty data, and the dirty cache lines are invalidated or overwritten in a ring buffer with new data to avoid the mandatory writeback to DRAM. Some of the new CXL protocols coming down the pipeline might improve this situation, but they also might not.
- drewg123 5y agoI've been looking at this. When using inline hardware kTLS offload, this is an amazing solution on paper. However, NVME speaks in 4KB blocks, but NICs speak in ~1400 byte MSS chunks. According to a colleague who knows the internal architecture of NVME drives well, making 1400b requests is a non-starter. So to use P2P DMA, you need some place to cache things. There turn out to be 3 places to cache things w/o going to the host: 1) On the NVME drive itself. This is known as controller memory buffer (CMB). Unfortunately, in order to get a CMB that's large enough to be useful, you need to buy gold-plated enterprise drives, which would be fine for a science experiment, but does not scale. 2) On the NIC. We use Mellanox CX6-DX NICs, and they have essentially no storage. 3) On something on the PCIe bus that has memory, like a GPU or custom "box of ram". The GPU would blow our thermal budget, and I can't think of a low cost, low-power "box of RAM" that can be attached at full speed to PCIe.
- magicalhippo 5y ago> However, NVME speaks in 4KB blocks, but NICs speak in ~1400 byte MSS chunks. Ah, I had imagined the NIC TLS stuff to be more stream based, clearly out of my comfort zone this stuff. > I can't think of a low cost, low-power "box of RAM" that can be attached at full speed to PCIe. Yeah, I mean some FPGA cards could probably do the trick, but would not be a low cost option and I guess not low power if you want full speed PCIe. Close but no cigar for p2pdma then...
- drewg123 5y agoWe've spoken to at least one small NIC vendor that puts ~48GB of RAM on their FPGA based smart nic. That let's them handle TCP rexmits without paying the price of re-DMA'ing data from the host to re-encrypt it. I wish the FPGA didn't blow our power budget, because their solution would be ideal for P2P DMA.
- lostmsu 5y agoCan't you buy custom RAM (e.g. GDDR or HBM)?
- shadilay 5y agoThis is called a smartNIC.