3 ms·

I think in general I agree. However the overhead numbers are exaggerated, and we should be fair with that. E.g. it was already mentioned that multiple UDP packe

I think in general I agree. However the overhead numbers are exaggerated, and we should be fair with that. E.g. it was already mentioned that multiple UDP packets can be transmitted via a single syscall, and reasonable implementations can make use of it. I haven't read the Quic spec (yet), so I don't know how much data can be aggregated without waiting for ACKs or interleaving other data - but if it's anything comparable to HTTP/2 then it should be configurable and support >= 64kB chunks. I also don't think a QUIC server would read the whole file into user-space at once - that's just a giant memory waste. Rather it would be streamed and chunks would get encrypted. That process requires of course an extra copy (likely even two for the unencrypted and encrypted version), but that's the same for all user-space file serving and encryption options and nothing new due to QUIC. For KTLS it would need to get investigated whether the kernel solution doesn't also perform a copy somewhere (I honestly don't know).

drewg123 8y ago
Of course it is not going to read the entire file at once. Having written the FreeBSD kernel TLS, I can assure you that there is no copy. Data is brought into the kernel via DMA from storage into a page in the VM page cache. When the IO is done, it is then encrypted into an connection-private page. That page is then sent and DMA'ed on the network adapter. So we have in the kernel tls case: - memory DMA to kernel mem from storage. - memory READ from kernel mem to read plaintext for crypto - memory write to another chunk of kernel mem to write encrypted data - memory DMA from kernel mem to NIC In the case where the NIC supports inline TLS offload, the middle 2 steps are skipped, and it devolves to essentially the unencrypted case. For QUIC you have: - memory DMA to kernel mem from storage - memory read from kernel mem via mmap - memory write to userspace mem to write encrypted data - memory read from userspace mem to copy to kernel - memory write to kernel mem - memory DMA from kernel mem to NIC So you go from 3 "copies" to 4 "copies", which increases memory bandwidth demands by 33%. Right now, we can just barely serve 100g from a Xeon-D because Intel limited the memory bandwidth to DDR4-2400. At an effective bandwidth limit of 60GB/sec, that's on the edge of being able to handle the kernel TLS data path. So even if everything else about QUIC was free, this extra memory copy from userspace would cut bandwidth by a third.
- Matthias247 8y ago
  Good to know. Thanks for the explanation and all the insights!

drewg123 8y ago

Of course it is not going to read the entire file at once. Having written the FreeBSD kernel TLS, I can assure you that there is no copy. Data is brought into the kernel via DMA from storage into a page in the VM page cache. When the IO is done, it is then encrypted into an connection-private page. That page is then sent and DMA'ed on the network adapter. So we have in the kernel tls case: - memory DMA to kernel mem from storage. - memory READ from kernel mem to read plaintext for crypto - memory write to another chunk of kernel mem to write encrypted data - memory DMA from kernel mem to NIC In the case where the NIC supports inline TLS offload, the middle 2 steps are skipped, and it devolves to essentially the unencrypted case. For QUIC you have: - memory DMA to kernel mem from storage - memory read from kernel mem via mmap - memory write to userspace mem to write encrypted data - memory read from userspace mem to copy to kernel - memory write to kernel mem - memory DMA from kernel mem to NIC So you go from 3 "copies" to 4 "copies", which increases memory bandwidth demands by 33%. Right now, we can just barely serve 100g from a Xeon-D because Intel limited the memory bandwidth to DDR4-2400. At an effective bandwidth limit of 60GB/sec, that's on the edge of being able to handle the kernel TLS data path. So even if everything else about QUIC was free, this extra memory copy from userspace would cut bandwidth by a third.

Matthias247 8y ago
Good to know. Thanks for the explanation and all the insights!