5 ms·

We've been running kTLS + SSL sendfile on FreeBSD at Netflix for the last 6 or 7 years. (We had local patches to nginx, before nginx did them "right", and 2 ve

We've been running kTLS + SSL sendfile on FreeBSD at Netflix for the last 6 or 7 years. (We had local patches to nginx, before nginx did them "right", and 2 versions of kTLS before the 2nd version was upstreamed to FreeBSD). The savings in terms of CPU use and memory BW are pretty substantial. Especially when you use a NIC which can do in-line kTLS offload, then things basically go back to pre-TLS costs because the buffers are not touched at all by the CPU. BTW, FreeBSD 14 supports cha-cha poly. But is far more CPU intensive than GCM, so I'd advise against using it.

cperciva 5y ago
Random question: Do you force Netflix clients onto the ciphers which are most efficient for the Netflix servers, or are there cases (I'm thinking mobile devices particularly) where it makes sense to use the ciphers which are most efficient for the clients?
- jchw 5y ago
  Pretty sure the best choice is basically always AES GCM because most modern chipsets can hardware offload that. Curious to hear the answer, though.
  - stragies 5y ago
    Genuine quesion: In 2022, is that still dependent on "Can I trust my (HW?)random number generator"?
    - jchw 5y ago
      I don’t believe so. AES is symmetric-key encryption; if you do it incorrectly, it doesn’t decrypt at all. The only place you can mess up, then, is in the mode, which I don’t believe is accelerated by most (any?) AES encryption instruction sets; instead, they mostly handle doing an individual round of encryption/decryption. Public key crypto systems seem like they would be much scarier to have hardware acceleration, though I’m sure if you broke it down to low level enough bits you could make it impossible for the hardware to “break” it (aside from, well, if it decided to just maliciously subsitute your code for its own. But it could do that without extension ISAs.)
      - CorrectHorseBat 5y ago
        How is the story any different for public key crypto?
        jchw 5y ago
        I am not a cryptography expert, so please take what I say with a grain of nacl. The parameters you choose for public key crypto systems can weaken the cryptographic properties of the result. For example, nonce reuse can single-handedly destroy your security. Similarly, with RSA, padding is crucial to security. Even if you write the extension ISAs such that they never choose parameters, there’s still probably some room for their to be broken output that seems to work.
      - amadvance 5y ago
        GCM is typically implemented in Intel assembler with the PCLMULQDQ instruction https://www.intel.com/content/dam/www/public/us/en/documents/software-support/enabling-high-performance-gcm.pdf https://www.intel.com/content/dam/www/public/us/en/documents...
    - netheril96 5y ago
      It of course depends on the random number generator. That is how you create the keys and sometimes the initialization vector. But AES itself is deterministic so it’s not possible to backdoor that part.
  - LinAGKar 5y ago
    x86 and ARMv8 do, but not ARMv7
    - mobilio 5y ago
      And one huge exception - RPi CPU.
- toast0 5y ago
  There are cases where it makes sense, but I'm not sure that mobile devices that are likely to be playing video is it. Chances are, they'll be burning more power on the screen backlight than the CPU to do AES vs something else (assuming they're not accelerated for AES). There's two sides to the argument, but easing the burden on servers lets one server serve more clients; it's easier to justify 2x the cost for crypto on the client than on the server, because most clients aren't bottlenecked on crypto and some servers are. (Of course, I'm usually a server engineer, so of course I want my servers to have less work ;) Choosing ciphers for ease of the client makes more sense, IMHO, when the client is really constrained, like a feature phone or tiny IoT things.
tinco 5y ago
Was CPU a bottleneck for your throughput, or did you just want some more headroom on those machines? I've been in the room with some web scale companies, but never was aware of any big excess TLS termination costs, but maybe I just didn't ask and they had some specialized hardware/software that I didn't know about.
- vasco 5y ago
  Not at netflix but at scale any savings in CPU usage translate directly into spend savings. If you have 100 machines at peak and autoscaling rules in place, you can generally assume that a 1% saving in CPU usage translates into one less machine running. You never get "headroom", you just have less machines running at any given time.
  - tinco 5y ago
    That's only true if CPU is the bottleneck, which is why I asked. I would expect network throughput to be the bottleneck, but maybe that's wrong nowadays with cheap 200gbit nics.
    - jfindley 5y ago
      Video streaming is a bit different from "normal" TLS web serving. For video streaming the network is much more likely to be the bottleneck, but traditional TLS web serving tends to send a lot of fairly small packets, and saturating even 40gbe with small packets of TLS is non-trivial. My experience is that at small (think less than 700 bytes or so) packet sizes, TLS overhead can be a very real cost as you scale up, and it's easy to hit "walls" where really you just need to buy more servers because the engineering cost of getting your servers to really saturate their NICs isn't worth it. How much of a big deal that is for you will depend a lot on exactly what sort of a thing you're serving, though.
      - drewg123 5y ago
        I'm curious why TLS packets are small in your experience. We use 16k TLS records, and send at whatever MSS the client has negotatied with us. Thats normally something "normal", like 1420-1460 bytes. Plus we use TSO, so the NIC sees large sends of up to 64KB.

cperciva 5y ago

Random question: Do you force Netflix clients onto the ciphers which are most efficient for the Netflix servers, or are there cases (I'm thinking mobile devices particularly) where it makes sense to use the ciphers which are most efficient for the clients?

jchw 5y ago
Pretty sure the best choice is basically always AES GCM because most modern chipsets can hardware offload that. Curious to hear the answer, though.
- stragies 5y ago
  Genuine quesion: In 2022, is that still dependent on "Can I trust my (HW?)random number generator"?
  - jchw 5y ago
    I don’t believe so. AES is symmetric-key encryption; if you do it incorrectly, it doesn’t decrypt at all. The only place you can mess up, then, is in the mode, which I don’t believe is accelerated by most (any?) AES encryption instruction sets; instead, they mostly handle doing an individual round of encryption/decryption. Public key crypto systems seem like they would be much scarier to have hardware acceleration, though I’m sure if you broke it down to low level enough bits you could make it impossible for the hardware to “break” it (aside from, well, if it decided to just maliciously subsitute your code for its own. But it could do that without extension ISAs.)
    - CorrectHorseBat 5y ago
      How is the story any different for public key crypto?
      - jchw 5y ago
        I am not a cryptography expert, so please take what I say with a grain of nacl. The parameters you choose for public key crypto systems can weaken the cryptographic properties of the result. For example, nonce reuse can single-handedly destroy your security. Similarly, with RSA, padding is crucial to security. Even if you write the extension ISAs such that they never choose parameters, there’s still probably some room for their to be broken output that seems to work.
    - amadvance 5y ago
      GCM is typically implemented in Intel assembler with the PCLMULQDQ instruction https://www.intel.com/content/dam/www/public/us/en/documents/software-support/enabling-high-performance-gcm.pdf https://www.intel.com/content/dam/www/public/us/en/documents...
  - netheril96 5y ago
    It of course depends on the random number generator. That is how you create the keys and sometimes the initialization vector. But AES itself is deterministic so it’s not possible to backdoor that part.
- LinAGKar 5y ago
  x86 and ARMv8 do, but not ARMv7
  - mobilio 5y ago
    And one huge exception - RPi CPU.
toast0 5y ago
There are cases where it makes sense, but I'm not sure that mobile devices that are likely to be playing video is it. Chances are, they'll be burning more power on the screen backlight than the CPU to do AES vs something else (assuming they're not accelerated for AES). There's two sides to the argument, but easing the burden on servers lets one server serve more clients; it's easier to justify 2x the cost for crypto on the client than on the server, because most clients aren't bottlenecked on crypto and some servers are. (Of course, I'm usually a server engineer, so of course I want my servers to have less work ;) Choosing ciphers for ease of the client makes more sense, IMHO, when the client is really constrained, like a feature phone or tiny IoT things.

tinco 5y ago

Was CPU a bottleneck for your throughput, or did you just want some more headroom on those machines? I've been in the room with some web scale companies, but never was aware of any big excess TLS termination costs, but maybe I just didn't ask and they had some specialized hardware/software that I didn't know about.

vasco 5y ago
Not at netflix but at scale any savings in CPU usage translate directly into spend savings. If you have 100 machines at peak and autoscaling rules in place, you can generally assume that a 1% saving in CPU usage translates into one less machine running. You never get "headroom", you just have less machines running at any given time.
- tinco 5y ago
  That's only true if CPU is the bottleneck, which is why I asked. I would expect network throughput to be the bottleneck, but maybe that's wrong nowadays with cheap 200gbit nics.
  - jfindley 5y ago
    Video streaming is a bit different from "normal" TLS web serving. For video streaming the network is much more likely to be the bottleneck, but traditional TLS web serving tends to send a lot of fairly small packets, and saturating even 40gbe with small packets of TLS is non-trivial. My experience is that at small (think less than 700 bytes or so) packet sizes, TLS overhead can be a very real cost as you scale up, and it's easy to hit "walls" where really you just need to buy more servers because the engineering cost of getting your servers to really saturate their NICs isn't worth it. How much of a big deal that is for you will depend a lot on exactly what sort of a thing you're serving, though.
    - drewg123 5y ago
      I'm curious why TLS packets are small in your experience. We use 16k TLS records, and send at whatever MSS the client has negotatied with us. Thats normally something "normal", like 1420-1460 bytes. Plus we use TSO, so the NIC sees large sends of up to 64KB.