3 ms·
Yes, without TLS offload, we're CPU limited. For example, on our 400GbE (4x100GbE) servers, we are CPU (really memory bandwidth) limited at ~240Gb/s without NI
by drewg123 3y ago
Yes, without TLS offload, we're CPU limited. For example, on our 400GbE (4x100GbE) servers, we are CPU (really memory bandwidth) limited at ~240Gb/s without NIC TLS offload. Using Mellanox CX6-DX NICs to move most of crypto onto the NIC increases the effective limit to ~375Gb/s.
We encrypt everything we can with TLS to protect the privacy of our members.
- jeffbee 3y agoThanks. I've read all the Netflix posts about the difficulty of moving 400gbps through a NUMA system. I imagined that if it was load-store limited then load-encrypt-store could potentially be "free".
- drewg123 3y agoThe happy path is storage ===>> RAM ==>> NIC, without CPU access. So there is basically a single memory write and a single memory read per byte. That's the non-TLS (and NIC TLS offload) path. Since TLS is per-connection and the page-cache is per-file, any software crypto needs to encrypt from a common source in the page cache to a per-client buffer (eg, it cannot happen in-place). So this introduces an extra memory read from the page cache and write to the per-client socket buffer. We use the non-temporal version of ISA-L, so as to write full cache lines into the socket buffer, and not pay for an extra memory read as part of a read-modify-write update of a partial cacheline. So software crypto basically doubles the memory bandwidth requirements over no TLS (or NIC TLS offload).