3 ms·

I've been looking at this. When using inline hardware kTLS offload, this is an amazing solution on paper. However, NVME speaks in 4KB blocks, but NICs speak i

I've been looking at this. When using inline hardware kTLS offload, this is an amazing solution on paper. However, NVME speaks in 4KB blocks, but NICs speak in ~1400 byte MSS chunks. According to a colleague who knows the internal architecture of NVME drives well, making 1400b requests is a non-starter. So to use P2P DMA, you need some place to cache things. There turn out to be 3 places to cache things w/o going to the host: 1) On the NVME drive itself. This is known as controller memory buffer (CMB). Unfortunately, in order to get a CMB that's large enough to be useful, you need to buy gold-plated enterprise drives, which would be fine for a science experiment, but does not scale. 2) On the NIC. We use Mellanox CX6-DX NICs, and they have essentially no storage. 3) On something on the PCIe bus that has memory, like a GPU or custom "box of ram". The GPU would blow our thermal budget, and I can't think of a low cost, low-power "box of RAM" that can be attached at full speed to PCIe.

magicalhippo 5y ago
> However, NVME speaks in 4KB blocks, but NICs speak in ~1400 byte MSS chunks. Ah, I had imagined the NIC TLS stuff to be more stream based, clearly out of my comfort zone this stuff. > I can't think of a low cost, low-power "box of RAM" that can be attached at full speed to PCIe. Yeah, I mean some FPGA cards could probably do the trick, but would not be a low cost option and I guess not low power if you want full speed PCIe. Close but no cigar for p2pdma then...
- drewg123 5y ago
  We've spoken to at least one small NIC vendor that puts ~48GB of RAM on their FPGA based smart nic. That let's them handle TCP rexmits without paying the price of re-DMA'ing data from the host to re-encrypt it. I wish the FPGA didn't blow our power budget, because their solution would be ideal for P2P DMA.
lostmsu 5y ago
Can't you buy custom RAM (e.g. GDDR or HBM)?

magicalhippo 5y ago

> However, NVME speaks in 4KB blocks, but NICs speak in ~1400 byte MSS chunks. Ah, I had imagined the NIC TLS stuff to be more stream based, clearly out of my comfort zone this stuff. > I can't think of a low cost, low-power "box of RAM" that can be attached at full speed to PCIe. Yeah, I mean some FPGA cards could probably do the trick, but would not be a low cost option and I guess not low power if you want full speed PCIe. Close but no cigar for p2pdma then...

drewg123 5y ago
We've spoken to at least one small NIC vendor that puts ~48GB of RAM on their FPGA based smart nic. That let's them handle TCP rexmits without paying the price of re-DMA'ing data from the host to re-encrypt it. I wish the FPGA didn't blow our power budget, because their solution would be ideal for P2P DMA.

lostmsu 5y ago

Can't you buy custom RAM (e.g. GDDR or HBM)?