3 ms·

The problem is that with current generation NICs and NVME drives, you need a buffer somewhere, and there is no place for the buffer. You need a buffer because

The problem is that with current generation NICs and NVME drives, you need a buffer somewhere, and there is no place for the buffer. You need a buffer because the NVME drive speaks in 4k chunks, and doing anything sub-4K is non optimal. So picture that TCP wants to send 2 segments (1448* 2 == 2896), possibly from a strange offset (say 7240, or 5 segments in). You'd be asking the NVME drive to read 2 4K blocks and pull 2896 bytes out of the middle. What you really want to do is read the first 12K and keep it buffered. With the current generation NICs, there is no place to buffer. With non gold-plated enterprise NVME, there is no place to buffer. NVME does have a thing called CMB, but the last time I searched, I could not find a drive with CMB that was large enough to be useful at a decent price. The alternative is to have a box of RAM sitting on the PCIe bus. That's pretty much a GPU, but GPUs are expensive, power-hungry, and unobtainable. So for now, its just not practical to do this.

a1369209993 5y ago
> So picture that TCP wants to send 2 segments (1448* 2 == 2896), possibly from a strange offset (say 7240, or 5 segments in). You'd be asking the NVME drive to read 2 4K blocks and pull 2896 bytes out of the middle. What you really want to do is read the first 12K and keep it buffered. The obvious solution here would be to just send less-than-maximum-size TCP segments. You have to support that to some extent anyway, to deal with the last segment of a non-multiple-of-1448 file, and to talk to things behind small MSSs. So send 4 1024-byte segments, or 3 segments of 1368+1368+1360 bytes (or whatever subdivision is technically convenient). If you can stream out 181 uint64s of data (from RAM, I assume?) in the first place, it shouldn't be that much harder to stream out 171 or 170. There may be technical reasons why that won't work (eg the hardware is double-buffered and only supports current-packet and next-packet), but it's not just a matter of "TCP segments are a irregular size", because TCP segments are variable sized.
- drewg123 5y ago
  That's the obvious solution. But we A/B test QoE metrics to death. Touching something so fundamental about TCP would surely hurt QoE. The last time we tried to coerce TCP into helping with a problem like this, we were dealing with a NIC that could not save state in the middle of TLS records. Eg, if TCP sent 2800 bytes, then tried to send another 2800 bytes, the NIC would need to re-DMA those first 2800 bytes so as to be able to encrypt the 2800 bytes that it was asked to send. So we made the TLS record size variable from 16k down to 4K, and forced TCP to always send a complete TLS record at a TLS record boundary so as to avoid having the NIC have to re-DMA the start of a TLS record. This was *MISERABLE* for QoE. It wasn't the NIC itself; enforcing those restrictions led to horrible QoE with any NIC. So no, I'm not gonna touch anything having to do with TCP :(
  - infogulch 5y ago
    Have you considered using a cypher mode like AES-CTR [0]? That might reduce the need to buffer because later packets are no longer dependent on the encryption result of previous packets. Of course that would only work if clients support CTR, and if the NIC supports it, and depends on how much you trust the encryption, but if it applies it could help a lot and eliminate the whole issue of retransmissions causing IO spikes to rebuild the TLS state. To be clear, I just wondered if such an encryption mode existed and found this after spending 2m googling it, so I have no idea what the adoption rate of such a cypher is or what the consensus is on its security. And for all I know you're already using it and it doesn't solve the problem because something else is forcing the NIC to rebuild state. [0] https://datatracker.ietf.org/doc/html/draft-ietf-tls-ctr-01#section-1 https://datatracker.ietf.org/doc/html/draft-ietf-tls-ctr-01#... > AES-CTR is capable of random access within the key stream. For DTLS, this implies that records can be processed out of order without dependency on packet arrival order, and also without keystream buffering
toast0 5y ago
> you need a buffer somewhere, and there is no place for the buffer. > The alternative is to have a box of RAM sitting on the PCIe bus. One thing I didn't think of until now is your 400 Gbps system uses effectively 32 PCI-e 4.0 lanes of Nic and 32 more lanes of storage (i think it's 64 lanes of 3.0 storage in system), and you mention elsewhere that your 800 Gbps system needs 64 lanes of NIC and about 64 lanes of storage, and that you're having a hard time finding that many lanes on a board that meets your requirements. You'd need more lanes for the PCI-e ram disk, even if you could find one more affordable and available than GPUs. Sounds like Mellanox (or someone) needs to add some ram to their NICs. (Or wait for PCI-e 5.0 and DDR5)

a1369209993 5y ago

> So picture that TCP wants to send 2 segments (1448* 2 == 2896), possibly from a strange offset (say 7240, or 5 segments in). You'd be asking the NVME drive to read 2 4K blocks and pull 2896 bytes out of the middle. What you really want to do is read the first 12K and keep it buffered. The obvious solution here would be to just send less-than-maximum-size TCP segments. You have to support that to some extent anyway, to deal with the last segment of a non-multiple-of-1448 file, and to talk to things behind small MSSs. So send 4 1024-byte segments, or 3 segments of 1368+1368+1360 bytes (or whatever subdivision is technically convenient). If you can stream out 181 uint64s of data (from RAM, I assume?) in the first place, it shouldn't be that much harder to stream out 171 or 170. There may be technical reasons why that won't work (eg the hardware is double-buffered and only supports current-packet and next-packet), but it's not just a matter of "TCP segments are a irregular size", because TCP segments are variable sized.

drewg123 5y ago
That's the obvious solution. But we A/B test QoE metrics to death. Touching something so fundamental about TCP would surely hurt QoE. The last time we tried to coerce TCP into helping with a problem like this, we were dealing with a NIC that could not save state in the middle of TLS records. Eg, if TCP sent 2800 bytes, then tried to send another 2800 bytes, the NIC would need to re-DMA those first 2800 bytes so as to be able to encrypt the 2800 bytes that it was asked to send. So we made the TLS record size variable from 16k down to 4K, and forced TCP to always send a complete TLS record at a TLS record boundary so as to avoid having the NIC have to re-DMA the start of a TLS record. This was *MISERABLE* for QoE. It wasn't the NIC itself; enforcing those restrictions led to horrible QoE with any NIC. So no, I'm not gonna touch anything having to do with TCP :(
- infogulch 5y ago
  Have you considered using a cypher mode like AES-CTR [0]? That might reduce the need to buffer because later packets are no longer dependent on the encryption result of previous packets. Of course that would only work if clients support CTR, and if the NIC supports it, and depends on how much you trust the encryption, but if it applies it could help a lot and eliminate the whole issue of retransmissions causing IO spikes to rebuild the TLS state. To be clear, I just wondered if such an encryption mode existed and found this after spending 2m googling it, so I have no idea what the adoption rate of such a cypher is or what the consensus is on its security. And for all I know you're already using it and it doesn't solve the problem because something else is forcing the NIC to rebuild state. [0] https://datatracker.ietf.org/doc/html/draft-ietf-tls-ctr-01#section-1 https://datatracker.ietf.org/doc/html/draft-ietf-tls-ctr-01#... > AES-CTR is capable of random access within the key stream. For DTLS, this implies that records can be processed out of order without dependency on packet arrival order, and also without keystream buffering

toast0 5y ago

> you need a buffer somewhere, and there is no place for the buffer. > The alternative is to have a box of RAM sitting on the PCIe bus. One thing I didn't think of until now is your 400 Gbps system uses effectively 32 PCI-e 4.0 lanes of Nic and 32 more lanes of storage (i think it's 64 lanes of 3.0 storage in system), and you mention elsewhere that your 800 Gbps system needs 64 lanes of NIC and about 64 lanes of storage, and that you're having a hard time finding that many lanes on a board that meets your requirements. You'd need more lanes for the PCI-e ram disk, even if you could find one more affordable and available than GPUs. Sounds like Mellanox (or someone) needs to add some ram to their NICs. (Or wait for PCI-e 5.0 and DDR5)