3 ms·
But would that be relevant? In other words, are CDN nodes CPU-limited? Or, does the bulk encryption absorb otherwise unused CPU time? I would have imagined that
by jeffbee 3y ago
But would that be relevant? In other words, are CDN nodes CPU-limited? Or, does the bulk encryption absorb otherwise unused CPU time? I would have imagined that most Netflix traffic is to clients that Netflix itself controls, like Roku apps or other first-party apps and not web browsers, so Netflix could choose against encryption for most streams.
- ComputerGuru 3y agoSo you're proposing that Netflix have two completely separate pipelines, one for browser traffic and the other for first-party traffic? With separate CDNs, separate delivery mechanisms, separate client-side algorithms, etc?
- jeffbee 3y agoWhat?
- drewg123 3y agoYes, without TLS offload, we're CPU limited. For example, on our 400GbE (4x100GbE) servers, we are CPU (really memory bandwidth) limited at ~240Gb/s without NIC TLS offload. Using Mellanox CX6-DX NICs to move most of crypto onto the NIC increases the effective limit to ~375Gb/s. We encrypt everything we can with TLS to protect the privacy of our members.
- jeffbee 3y agoThanks. I've read all the Netflix posts about the difficulty of moving 400gbps through a NUMA system. I imagined that if it was load-store limited then load-encrypt-store could potentially be "free".
- drewg123 3y agoThe happy path is storage ===>> RAM ==>> NIC, without CPU access. So there is basically a single memory write and a single memory read per byte. That's the non-TLS (and NIC TLS offload) path. Since TLS is per-connection and the page-cache is per-file, any software crypto needs to encrypt from a common source in the page cache to a per-client buffer (eg, it cannot happen in-place). So this introduces an extra memory read from the page cache and write to the per-client socket buffer. We use the non-temporal version of ISA-L, so as to write full cache lines into the socket buffer, and not pay for an extra memory read as part of a read-modify-write update of a partial cacheline. So software crypto basically doubles the memory bandwidth requirements over no TLS (or NIC TLS offload).