4 ms·

That's the obvious solution. But we A/B test QoE metrics to death. Touching something so fundamental about TCP would surely hurt QoE. The last time we tried

That's the obvious solution. But we A/B test QoE metrics to death. Touching something so fundamental about TCP would surely hurt QoE. The last time we tried to coerce TCP into helping with a problem like this, we were dealing with a NIC that could not save state in the middle of TLS records. Eg, if TCP sent 2800 bytes, then tried to send another 2800 bytes, the NIC would need to re-DMA those first 2800 bytes so as to be able to encrypt the 2800 bytes that it was asked to send. So we made the TLS record size variable from 16k down to 4K, and forced TCP to always send a complete TLS record at a TLS record boundary so as to avoid having the NIC have to re-DMA the start of a TLS record. This was *MISERABLE* for QoE. It wasn't the NIC itself; enforcing those restrictions led to horrible QoE with any NIC. So no, I'm not gonna touch anything having to do with TCP :(

infogulch 5y ago
Have you considered using a cypher mode like AES-CTR [0]? That might reduce the need to buffer because later packets are no longer dependent on the encryption result of previous packets. Of course that would only work if clients support CTR, and if the NIC supports it, and depends on how much you trust the encryption, but if it applies it could help a lot and eliminate the whole issue of retransmissions causing IO spikes to rebuild the TLS state. To be clear, I just wondered if such an encryption mode existed and found this after spending 2m googling it, so I have no idea what the adoption rate of such a cypher is or what the consensus is on its security. And for all I know you're already using it and it doesn't solve the problem because something else is forcing the NIC to rebuild state. [0] https://datatracker.ietf.org/doc/html/draft-ietf-tls-ctr-01#section-1 https://datatracker.ietf.org/doc/html/draft-ietf-tls-ctr-01#... > AES-CTR is capable of random access within the key stream. For DTLS, this implies that records can be processed out of order without dependency on packet arrival order, and also without keystream buffering

infogulch 5y ago

Have you considered using a cypher mode like AES-CTR [0]? That might reduce the need to buffer because later packets are no longer dependent on the encryption result of previous packets. Of course that would only work if clients support CTR, and if the NIC supports it, and depends on how much you trust the encryption, but if it applies it could help a lot and eliminate the whole issue of retransmissions causing IO spikes to rebuild the TLS state. To be clear, I just wondered if such an encryption mode existed and found this after spending 2m googling it, so I have no idea what the adoption rate of such a cypher is or what the consensus is on its security. And for all I know you're already using it and it doesn't solve the problem because something else is forcing the NIC to rebuild state. [0] https://datatracker.ietf.org/doc/html/draft-ietf-tls-ctr-01#section-1 https://datatracker.ietf.org/doc/html/draft-ietf-tls-ctr-01#... > AES-CTR is capable of random access within the key stream. For DTLS, this implies that records can be processed out of order without dependency on packet arrival order, and also without keystream buffering