3 ms·

Was CPU a bottleneck for your throughput, or did you just want some more headroom on those machines? I've been in the room with some web scale companies, but n

by tinco 5y ago

Was CPU a bottleneck for your throughput, or did you just want some more headroom on those machines? I've been in the room with some web scale companies, but never was aware of any big excess TLS termination costs, but maybe I just didn't ask and they had some specialized hardware/software that I didn't know about.

vasco 5y ago
Not at netflix but at scale any savings in CPU usage translate directly into spend savings. If you have 100 machines at peak and autoscaling rules in place, you can generally assume that a 1% saving in CPU usage translates into one less machine running. You never get "headroom", you just have less machines running at any given time.
- tinco 5y ago
  That's only true if CPU is the bottleneck, which is why I asked. I would expect network throughput to be the bottleneck, but maybe that's wrong nowadays with cheap 200gbit nics.
  - jfindley 5y ago
    Video streaming is a bit different from "normal" TLS web serving. For video streaming the network is much more likely to be the bottleneck, but traditional TLS web serving tends to send a lot of fairly small packets, and saturating even 40gbe with small packets of TLS is non-trivial. My experience is that at small (think less than 700 bytes or so) packet sizes, TLS overhead can be a very real cost as you scale up, and it's easy to hit "walls" where really you just need to buy more servers because the engineering cost of getting your servers to really saturate their NICs isn't worth it. How much of a big deal that is for you will depend a lot on exactly what sort of a thing you're serving, though.
    - drewg123 5y ago
      I'm curious why TLS packets are small in your experience. We use 16k TLS records, and send at whatever MSS the client has negotatied with us. Thats normally something "normal", like 1420-1460 bytes. Plus we use TSO, so the NIC sees large sends of up to 64KB.
toast0 5y ago
I'm not at Netflix, just summarizing. Check out the Netflix presentations like [1]; at 400gbps, the bottleneck is RAM and NUMA fabric. Doing bulk crypto on the CPU requires double the RAM bandwidth, and that kills the throughput regardless of the cost of cipher computation. TLS termination costs are a large factor for Netflix because their CDN appliances want to have as much throughput as possible in a small space. At 'web scale', it wouldn't be a big deal to be happy with 100gbps througput and use 4 boxes; but getting ISPs to install more boxes is hard, it's better to get one box to work as fast as possible. Personally, I've seen it be an issue; dual xeon 2690 v1 or maybe v2 couldn't push 10G via TLS, but could via plain text. But the v3 chips had better acceleration and could do 2x10G no problem. I never got access to more than 2x10G networking, so no idea what the limit was. That was under managed hosting (dedicated bare metal), so no ability to do networking beyond what was offered. At Facebook, they tended to use smaller servers and more of them, and I didn't deal with TLS as much. [1] https://people.freebsd.org/~gallatin/talks/euro2021.pdf https://people.freebsd.org/~gallatin/talks/euro2021.pdf
- drewg123 5y ago
  The limit for FreeBSD software kTLS with our workload is a little over 100Gb/s (using a 2x100G NIC) on a 2697a v4. I think most people only get 25-30Gb/s out of the same CPU when not using kTLS.
drewg123 5y ago
The bottleneck is both CPU and memory bandwidth. Doing kTLS in software (or in a lookaside accelerator, like QAT) doubles memory bandwidth requirements because something (CPU or accelerator hardware) needs to read the buffer to be encrypted, and write the result into a new buffer. And doing TLS in userspace, rather than kTLS is even worse, because it disables sendfile. That means you now have extra copies across the user/kernel boundary.

by tinco 5y ago

vasco 5y ago

Not at netflix but at scale any savings in CPU usage translate directly into spend savings. If you have 100 machines at peak and autoscaling rules in place, you can generally assume that a 1% saving in CPU usage translates into one less machine running. You never get "headroom", you just have less machines running at any given time.

tinco 5y ago
That's only true if CPU is the bottleneck, which is why I asked. I would expect network throughput to be the bottleneck, but maybe that's wrong nowadays with cheap 200gbit nics.
- jfindley 5y ago
  Video streaming is a bit different from "normal" TLS web serving. For video streaming the network is much more likely to be the bottleneck, but traditional TLS web serving tends to send a lot of fairly small packets, and saturating even 40gbe with small packets of TLS is non-trivial. My experience is that at small (think less than 700 bytes or so) packet sizes, TLS overhead can be a very real cost as you scale up, and it's easy to hit "walls" where really you just need to buy more servers because the engineering cost of getting your servers to really saturate their NICs isn't worth it. How much of a big deal that is for you will depend a lot on exactly what sort of a thing you're serving, though.
  - drewg123 5y ago
    I'm curious why TLS packets are small in your experience. We use 16k TLS records, and send at whatever MSS the client has negotatied with us. Thats normally something "normal", like 1420-1460 bytes. Plus we use TSO, so the NIC sees large sends of up to 64KB.

toast0 5y ago

I'm not at Netflix, just summarizing. Check out the Netflix presentations like [1]; at 400gbps, the bottleneck is RAM and NUMA fabric. Doing bulk crypto on the CPU requires double the RAM bandwidth, and that kills the throughput regardless of the cost of cipher computation. TLS termination costs are a large factor for Netflix because their CDN appliances want to have as much throughput as possible in a small space. At 'web scale', it wouldn't be a big deal to be happy with 100gbps througput and use 4 boxes; but getting ISPs to install more boxes is hard, it's better to get one box to work as fast as possible. Personally, I've seen it be an issue; dual xeon 2690 v1 or maybe v2 couldn't push 10G via TLS, but could via plain text. But the v3 chips had better acceleration and could do 2x10G no problem. I never got access to more than 2x10G networking, so no idea what the limit was. That was under managed hosting (dedicated bare metal), so no ability to do networking beyond what was offered. At Facebook, they tended to use smaller servers and more of them, and I didn't deal with TLS as much. [1] https://people.freebsd.org/~gallatin/talks/euro2021.pdf https://people.freebsd.org/~gallatin/talks/euro2021.pdf

drewg123 5y ago
The limit for FreeBSD software kTLS with our workload is a little over 100Gb/s (using a 2x100G NIC) on a 2697a v4. I think most people only get 25-30Gb/s out of the same CPU when not using kTLS.

drewg123 5y ago

The bottleneck is both CPU and memory bandwidth. Doing kTLS in software (or in a lookaside accelerator, like QAT) doubles memory bandwidth requirements because something (CPU or accelerator hardware) needs to read the buffer to be encrypted, and write the result into a new buffer. And doing TLS in userspace, rather than kTLS is even worse, because it disables sendfile. That means you now have extra copies across the user/kernel boundary.