3 ms·

> So picture that TCP wants to send 2 segments (1448* 2 == 2896), possibly from a strange offset (say 7240, or 5 segments in). You'd be asking the NVME drive to

by a1369209993 5y ago

> So picture that TCP wants to send 2 segments (1448* 2 == 2896), possibly from a strange offset (say 7240, or 5 segments in). You'd be asking the NVME drive to read 2 4K blocks and pull 2896 bytes out of the middle. What you really want to do is read the first 12K and keep it buffered. The obvious solution here would be to just send less-than-maximum-size TCP segments. You have to support that to some extent anyway, to deal with the last segment of a non-multiple-of-1448 file, and to talk to things behind small MSSs. So send 4 1024-byte segments, or 3 segments of 1368+1368+1360 bytes (or whatever subdivision is technically convenient). If you can stream out 181 uint64s of data (from RAM, I assume?) in the first place, it shouldn't be that much harder to stream out 171 or 170. There may be technical reasons why that won't work (eg the hardware is double-buffered and only supports current-packet and next-packet), but it's not just a matter of "TCP segments are a irregular size", because TCP segments are variable sized.

drewg123 5y ago
That's the obvious solution. But we A/B test QoE metrics to death. Touching something so fundamental about TCP would surely hurt QoE. The last time we tried to coerce TCP into helping with a problem like this, we were dealing with a NIC that could not save state in the middle of TLS records. Eg, if TCP sent 2800 bytes, then tried to send another 2800 bytes, the NIC would need to re-DMA those first 2800 bytes so as to be able to encrypt the 2800 bytes that it was asked to send. So we made the TLS record size variable from 16k down to 4K, and forced TCP to always send a complete TLS record at a TLS record boundary so as to avoid having the NIC have to re-DMA the start of a TLS record. This was *MISERABLE* for QoE. It wasn't the NIC itself; enforcing those restrictions led to horrible QoE with any NIC. So no, I'm not gonna touch anything having to do with TCP :(
- infogulch 5y ago
  Have you considered using a cypher mode like AES-CTR [0]? That might reduce the need to buffer because later packets are no longer dependent on the encryption result of previous packets. Of course that would only work if clients support CTR, and if the NIC supports it, and depends on how much you trust the encryption, but if it applies it could help a lot and eliminate the whole issue of retransmissions causing IO spikes to rebuild the TLS state. To be clear, I just wondered if such an encryption mode existed and found this after spending 2m googling it, so I have no idea what the adoption rate of such a cypher is or what the consensus is on its security. And for all I know you're already using it and it doesn't solve the problem because something else is forcing the NIC to rebuild state. [0] https://datatracker.ietf.org/doc/html/draft-ietf-tls-ctr-01#section-1 https://datatracker.ietf.org/doc/html/draft-ietf-tls-ctr-01#... > AES-CTR is capable of random access within the key stream. For DTLS, this implies that records can be processed out of order without dependency on packet arrival order, and also without keystream buffering

drewg123 5y ago

That's the obvious solution. But we A/B test QoE metrics to death. Touching something so fundamental about TCP would surely hurt QoE. The last time we tried to coerce TCP into helping with a problem like this, we were dealing with a NIC that could not save state in the middle of TLS records. Eg, if TCP sent 2800 bytes, then tried to send another 2800 bytes, the NIC would need to re-DMA those first 2800 bytes so as to be able to encrypt the 2800 bytes that it was asked to send. So we made the TLS record size variable from 16k down to 4K, and forced TCP to always send a complete TLS record at a TLS record boundary so as to avoid having the NIC have to re-DMA the start of a TLS record. This was *MISERABLE* for QoE. It wasn't the NIC itself; enforcing those restrictions led to horrible QoE with any NIC. So no, I'm not gonna touch anything having to do with TCP :(

infogulch 5y ago
Have you considered using a cypher mode like AES-CTR [0]? That might reduce the need to buffer because later packets are no longer dependent on the encryption result of previous packets. Of course that would only work if clients support CTR, and if the NIC supports it, and depends on how much you trust the encryption, but if it applies it could help a lot and eliminate the whole issue of retransmissions causing IO spikes to rebuild the TLS state. To be clear, I just wondered if such an encryption mode existed and found this after spending 2m googling it, so I have no idea what the adoption rate of such a cypher is or what the consensus is on its security. And for all I know you're already using it and it doesn't solve the problem because something else is forcing the NIC to rebuild state. [0] https://datatracker.ietf.org/doc/html/draft-ietf-tls-ctr-01#section-1 https://datatracker.ietf.org/doc/html/draft-ietf-tls-ctr-01#... > AES-CTR is capable of random access within the key stream. For DTLS, this implies that records can be processed out of order without dependency on packet arrival order, and also without keystream buffering