6 ms·
Any thoughts on Chelsio TLS offload?
by wmf 9y ago
Any thoughts on Chelsio TLS offload?
- kev009 9y agoI'm starting to test it in co-processor mode. The main limit is.. Intel is dragging their feet on PCIe bandwidth (probably to curtail GPGPU a bit), and you need a lot of BW for co-processor up and down and then back out the pipe. But it's enough for my needs at the moment. https://www.chelsio.com/wp-content/uploads/resources/T6-100G-Coprocessor-Linux.pdf https://www.chelsio.com/wp-content/uploads/resources/T6-100G...
- drewg123 9y agoAt least for broadwell based Xeons with just 2 memory controllers, our main bottleneck is memory bandwidth. And co-processor cards haven't really helped us much. The same memory accesses need to happen. The difference is that with a co-processor, they're done via DMA rather than via memory reads and writes. Intel has gotten so good with crypto in the recent chips that, at least for GCM, we didn't find much advantage to using add-in accelerators.
- shaklee3 9y agoHave you considered Skylake? The 6 memory channels should easily put you above 100Gbps now.
- drewg123 9y agoIt sounds very promising. In-line TLS is the holy grail for us, since it would cut our memory bandwidth usage by over 40%, and memory bandwidth is our biggest bottleneck. However, I'm not very bullish on just plain crypto accelerator mode. We've tried other vendor's accelerators, and while they save some CPU, memory bandwidth is the real issue for us. The data still needs to be encrypted, so it still needs to be read from memory by DMA rather than a CPU read, and written via a DMA write rather than a CPU write.
- FullyFunctional 9y ago(BTW: Huge kudos to Netflix for driving vendors to support FreeBSD) Unfortunately exposing my nativity on both NICs and TLS, I've never understood why NICs couldn't support the crypto in a way that it can just send the packet straight to the wire after encryption. Is there some deep (unfixable) reason in the TLS/TCP protocol or just a lack of foresight in the NIC design?
- wmf 9y agowhy NICs couldn't support the crypto in a way that it can just send the packet straight to the wire after encryption. That's how the Chelsio T6 works. On the transmit path the NIC needs to perform encryption, TLS framing, and TCP/IP/Ethernet segmentation in that order; on receive it needs to perform TCP reordering and such before decrypting. Most NICs just don't want to add that much functionality.
- FullyFunctional 9y agoThanks, that's looks pretty good. I wonder if it will work with TLS 1.3 as well (from my limited understanding it looks like it might). Here's dreaming of a future where ED25519/Chacha20/Poly1305 is supported in hardware (it's much cheaper for clients that doesn't have AES so I want servers to use it).
- drewg123 9y agoWhat makes in-line TLS difficult is that the NIC has to understand the TLS framing and the TCP segmentation for TSO. On top of that, what do you do if you have a re-transmission. The NIC has to re-download the entire 16KB TLS frame to re-send just a 1448b portion of it. So there are some challenges in terms of being able to find the entire TLS record, making sure to reference the entire thing so that it is all still there, etc. The "1 mbuf per TLS record" work I've done will make that a bit simpler, I hope.
- baybal2 9y ago>download the entire 16KB TLS frame to re-send just a 1448b portion of it. The bigger thing, is that TLS is far from ideal for video content, and TCP transmissions in general. It is much better to incorporate a degree of redundant coding, and tolerance to lost packets than rely on re-transmission. As you deal with a real-time content, there is a no real need for flow control. You either get packets in time, or your video feed cuts of. Another moment, for any genuinely live transmission, multicast beats down everything for efficiency, but why it is so hard to run multicast over open Internet is a discussion on its own.