5 ms·

The point they they seem to omit is that TLS roughly doubles the cost to serve static content, assuming no hardware offloads are used. Roughly 40% of our CPU t

The point they they seem to omit is that TLS roughly doubles the cost to serve static content, assuming no hardware offloads are used. Roughly 40% of our CPU time on the Netflix CDN (on nodes w/o TLS offload NICs) is spent doing TLS crypto. I'm not saying I'm against TLS, far from it. But it does carry costs, and I'm surprised somebody so passionately against TLS would omit this argument.

ComputerGuru 3y ago
Probably because for static text sites the compute cost of TLS is negligible (and we're largely talking about either small personal blogs or the long tail of historic content, so traffic is already assumed to be minimal). In your case, TLS is rather unfortunate since you can just include a crypto hash (or merkle tree or whatever) of the payload in your HTTPS control connection, then just load your video payloads over HTTP and check for a mismatch client-side. Basically, Google is saying they don't trust "you" to get it right. The bigger problem with TLS is that it adds so much brittleness and complexity (do you know how many times I've broken ACME/LE python dependencies by accident?) and increases the time-to-first-byte for so many sites, especially if you're not getting your hands dirty with nginx and OpenSSL internals. (Even if you are tuning all the right knobs, one thing we didn't realize was that at some point using an EV SSL certificate (before Chrome removed the UI benefits of doing so) began causing a massive increase in TTFB that there doesn't (didn't?) seem to be any way to avoid, so we ended up moving to regular TLS/SSL certs w/ Let's Encrypt not for the free cost of entry but for the reduced initial connection time. I can't remember the exact details now as it's been some years, but I think with EV SSL there's a secondary cert or revocation lookup to a remote url even if you have correctly configured ocsp stapling.)
jeffbee 3y ago
But would that be relevant? In other words, are CDN nodes CPU-limited? Or, does the bulk encryption absorb otherwise unused CPU time? I would have imagined that most Netflix traffic is to clients that Netflix itself controls, like Roku apps or other first-party apps and not web browsers, so Netflix could choose against encryption for most streams.
- ComputerGuru 3y ago
  So you're proposing that Netflix have two completely separate pipelines, one for browser traffic and the other for first-party traffic? With separate CDNs, separate delivery mechanisms, separate client-side algorithms, etc?
  - jeffbee 3y ago
    What?
- drewg123 3y ago
  Yes, without TLS offload, we're CPU limited. For example, on our 400GbE (4x100GbE) servers, we are CPU (really memory bandwidth) limited at ~240Gb/s without NIC TLS offload. Using Mellanox CX6-DX NICs to move most of crypto onto the NIC increases the effective limit to ~375Gb/s. We encrypt everything we can with TLS to protect the privacy of our members.
  - jeffbee 3y ago
    Thanks. I've read all the Netflix posts about the difficulty of moving 400gbps through a NUMA system. I imagined that if it was load-store limited then load-encrypt-store could potentially be "free".
    - drewg123 3y ago
      The happy path is storage ===>> RAM ==>> NIC, without CPU access. So there is basically a single memory write and a single memory read per byte. That's the non-TLS (and NIC TLS offload) path. Since TLS is per-connection and the page-cache is per-file, any software crypto needs to encrypt from a common source in the page cache to a per-client buffer (eg, it cannot happen in-place). So this introduces an extra memory read from the page cache and write to the per-client socket buffer. We use the non-temporal version of ISA-L, so as to write full cache lines into the socket buffer, and not pay for an extra memory read as part of a read-modify-write update of a partial cacheline. So software crypto basically doubles the memory bandwidth requirements over no TLS (or NIC TLS offload).
miyuru 3y ago
TLS lowers the throughput on thin clients too. From the MicroMirror Project. https://social.afront.org/@kwf/110499535819579680 https://social.afront.org/@kwf/110499535819579680
nektro 3y ago
this is such an old misconception

ComputerGuru 3y ago

Probably because for static text sites the compute cost of TLS is negligible (and we're largely talking about either small personal blogs or the long tail of historic content, so traffic is already assumed to be minimal). In your case, TLS is rather unfortunate since you can just include a crypto hash (or merkle tree or whatever) of the payload in your HTTPS control connection, then just load your video payloads over HTTP and check for a mismatch client-side. Basically, Google is saying they don't trust "you" to get it right. The bigger problem with TLS is that it adds so much brittleness and complexity (do you know how many times I've broken ACME/LE python dependencies by accident?) and increases the time-to-first-byte for so many sites, especially if you're not getting your hands dirty with nginx and OpenSSL internals. (Even if you are tuning all the right knobs, one thing we didn't realize was that at some point using an EV SSL certificate (before Chrome removed the UI benefits of doing so) began causing a massive increase in TTFB that there doesn't (didn't?) seem to be any way to avoid, so we ended up moving to regular TLS/SSL certs w/ Let's Encrypt not for the free cost of entry but for the reduced initial connection time. I can't remember the exact details now as it's been some years, but I think with EV SSL there's a secondary cert or revocation lookup to a remote url even if you have correctly configured ocsp stapling.)

jeffbee 3y ago

But would that be relevant? In other words, are CDN nodes CPU-limited? Or, does the bulk encryption absorb otherwise unused CPU time? I would have imagined that most Netflix traffic is to clients that Netflix itself controls, like Roku apps or other first-party apps and not web browsers, so Netflix could choose against encryption for most streams.

ComputerGuru 3y ago
So you're proposing that Netflix have two completely separate pipelines, one for browser traffic and the other for first-party traffic? With separate CDNs, separate delivery mechanisms, separate client-side algorithms, etc?
- jeffbee 3y ago
  What?
drewg123 3y ago
Yes, without TLS offload, we're CPU limited. For example, on our 400GbE (4x100GbE) servers, we are CPU (really memory bandwidth) limited at ~240Gb/s without NIC TLS offload. Using Mellanox CX6-DX NICs to move most of crypto onto the NIC increases the effective limit to ~375Gb/s. We encrypt everything we can with TLS to protect the privacy of our members.
- jeffbee 3y ago
  Thanks. I've read all the Netflix posts about the difficulty of moving 400gbps through a NUMA system. I imagined that if it was load-store limited then load-encrypt-store could potentially be "free".
  - drewg123 3y ago
    The happy path is storage ===>> RAM ==>> NIC, without CPU access. So there is basically a single memory write and a single memory read per byte. That's the non-TLS (and NIC TLS offload) path. Since TLS is per-connection and the page-cache is per-file, any software crypto needs to encrypt from a common source in the page cache to a per-client buffer (eg, it cannot happen in-place). So this introduces an extra memory read from the page cache and write to the per-client socket buffer. We use the non-temporal version of ISA-L, so as to write full cache lines into the socket buffer, and not pay for an extra memory read as part of a read-modify-write update of a partial cacheline. So software crypto basically doubles the memory bandwidth requirements over no TLS (or NIC TLS offload).

miyuru 3y ago

TLS lowers the throughput on thin clients too. From the MicroMirror Project. https://social.afront.org/@kwf/110499535819579680 https://social.afront.org/@kwf/110499535819579680

nektro 3y ago

this is such an old misconception