7 ms·

Improving NGINX Performance with Kernel TLS and SSL_sendfile

156 points by limoce 5y ago

schoen 5y ago
(2021)
- KarlKemp 5y ago
  November of 2021, to be exact. If you want to tag this article, you will have to tag others with (2022) sooner than one might expect.
  - 1500100900 5y ago
    There's so much churn on hacker news that everything gets old after a couple of nanoseconds. So I think it would be best to just add a field for a 128-bit timestamp of the original submission and present it with high precision.
rascul 5y ago
> Alpine Linux 3.11–3.14 – Kernel is built with the CONFIG_TLS=n option, which disables building kTLS as a module or as part of the kernel. I wonder if this is still the case with 3.15? Edit: I figured I could check for myself. I don't know for sure what the default kernel package is, but there apparently is a linux-lts package. After installing this package, it leaves a config-lts file in /boot which, when grepped, returns: # CONFIG_TLS is not set The more I learn about Alpine (and musl), the more I don't want to use them. It appears as if I have an inherent performance penalty serving https web sites with nginx when I do it from Alpine.
- limoce 5y ago
  What if I'm running NGINX inside a Alpine-based docker container, but the host OS is Ubuntu? I guess the kernel is okay but OpenSSL provided by Alpine is not ready for kTLS, so NGINX will not enable kTLS as well.
  - rascul 5y ago
    I didn't consider containers. I mostly run Alpine in virtual machines.
    - rascul 5y ago
      Hah I just realized I could check the kernel config in a vm. Earlier I looked for a kernel package to install in a container to see what the config was. Maybe I've been drinking too much. Anyway, in one of my Alpine virtual machines, /boot/config-virt has CONFIG_TLS unset.
      - bigbizisverywyz 5y ago
        The article actually mentions Alpine as one of the dist. where it's not supported by default: >The following OSs do not support kTLS, for the indicated reason: >Alpine Linux 3.11–3.14 – Kernel is built with the CONFIG_TLS=n option, which disables building kTLS as a module or as part of the kernel. and even recommends building OpenSSL and Nginx 3.0 yourself anyway, so looks like it will be a while before this might be available out-of-the-box for most major dists. But of course everything is OSS so you can DIY if you don't mind getting some ./configure under your fingernails :)
- Shared404 5y ago
  The way I see it is that it's just trade-off's to be chosen between. I like alpine because it's simple enough even for me to wrap my head around and understand what's going on. None of what I'm serving is high traffic or complex enough for this to matter to my usecase - and I suspect this applies to many people's situation. I appreciate musl/alpine for their stability and simplicity I suppose, and a bit of performance is an OK price to pay in my mind.
- stormbrew 5y ago
  > It appears as if I have an inherent performance penalty serving https web sites with nginx when I do it from Alpine. This is a weirdly alarmist take on this? If you're trying to use bleeding edge kernel features, which this basically is, you should probably feel comfortable using an alternative kernel because odds are pretty good you're gonna have to update sooner rather than later for some bug fix or other. It's just not really reasonable to expect all distros to enable all kernel flags all the time, a lot of them are not really proven safe or secure. Especially when they're new.
  - rascul 5y ago
    > This is a weirdly alarmist take on this It's an observation. I didn't intend for it to be alarmist, if you interpreted it that way then perhaps I could have worded it differently. > If you're trying to use bleeding edge kernel features, which this basically is It's been in the kernel since 2017 (the article noted kernel 4.13 which was released 2017 when I looked it up). That doesn't seem very bleeding edge to me. > It's just not really reasonable to expect all distros to enable all kernel flags all the time Of course. And I've already been considering moving away from Alpine for at least some use cases, and this can lead me to use move away for more use cases.
    - stormbrew 5y ago
      > It's an observation. I didn't intend for it to be alarmist, if you interpreted it that way then perhaps I could have worded it differently. Alarmist is perhaps the wrong word. What I mean is that this is a very strange and high bar for choosing a distro. You aren't "suffering a penalty by using alpine," you're being a beta tester by using a non-LTS ubuntu with a bunch of random flags on or whatever. You can also just.. use a different kernel version with alpine (or whatever distro), no one's stopping you. > It's been in the kernel since 2017 (the article noted kernel 4.13 which was released 2017 when I looked it up). That doesn't seem very bleeding edge to me. You can't actually use the version in 4.13 though, you need at least 4.17, because apparently that's what openssl 3.0.0 requires. Now 4.17 has been around for a while too! But you also need openssl 3.0.0 to make practical use of it, and that's only been out since sept 2021. And also had a massive number of breaking changes. And then you have to be using a newer kernel version than that to get tlsv3 ciphers apparently. Looks like somewhere around 5.10, though it doesn't explicitly say in the article afaict. If you don't use that then maybe you're gaining some speed but you're also downgrading your security. And then you need to use bleeding edge nginx and manually compile that against openssl3. So yeah. It's technically been there for years. But in practical terms no one (or very few people at least) have been using it in anger until the last few months. A new syscall in linux is "bleeding edge" for a while. Honestly the kernel is the least of your concerns here.
- _ikke_ 5y ago
  A feature request to enable it: https://gitlab.alpinelinux.org/alpine/aports/-/issues/13664 https://gitlab.alpinelinux.org/alpine/aports/-/issues/13664
winrid 5y ago
This would make Nchan even faster, neat.
- WJW 5y ago
  How so? According to the article this works only for serving static files. Nchan, cool as it is, has nothing to do with static files.
  - winrid 4y ago
    Ah, I see, I merely skimmed. Thanks for the correction.
drewg123 5y ago
We've been running kTLS + SSL sendfile on FreeBSD at Netflix for the last 6 or 7 years. (We had local patches to nginx, before nginx did them "right", and 2 versions of kTLS before the 2nd version was upstreamed to FreeBSD). The savings in terms of CPU use and memory BW are pretty substantial. Especially when you use a NIC which can do in-line kTLS offload, then things basically go back to pre-TLS costs because the buffers are not touched at all by the CPU. BTW, FreeBSD 14 supports cha-cha poly. But is far more CPU intensive than GCM, so I'd advise against using it.
- cperciva 5y ago
  Random question: Do you force Netflix clients onto the ciphers which are most efficient for the Netflix servers, or are there cases (I'm thinking mobile devices particularly) where it makes sense to use the ciphers which are most efficient for the clients?
  - jchw 5y ago
    Pretty sure the best choice is basically always AES GCM because most modern chipsets can hardware offload that. Curious to hear the answer, though.
    - stragies 5y ago
      Genuine quesion: In 2022, is that still dependent on "Can I trust my (HW?)random number generator"?
      - jchw 5y ago
        I don’t believe so. AES is symmetric-key encryption; if you do it incorrectly, it doesn’t decrypt at all. The only place you can mess up, then, is in the mode, which I don’t believe is accelerated by most (any?) AES encryption instruction sets; instead, they mostly handle doing an individual round of encryption/decryption. Public key crypto systems seem like they would be much scarier to have hardware acceleration, though I’m sure if you broke it down to low level enough bits you could make it impossible for the hardware to “break” it (aside from, well, if it decided to just maliciously subsitute your code for its own. But it could do that without extension ISAs.)
vsh_05 5y ago
Aissen 5y ago
Just a note to anyone wanting to use kTLS: make sure to benchmark it first, like in the article. Depending on the CPU architecture, it might even be slower than plain userspace TLS. Also, while the tx side has seen lots of investment (from CDN companies/owners), the receive side usually comes later. For instance, it's not supported for TLS 1.3 in openssl (although there's an open PR).
georgia_peach 5y ago
How long before we push everything into the kernel?
- CyberRabbi 5y ago
  Generally the kernel is involved when it comes to making use of hardware. Specialized hardware emerges when widely applicable bottlenecks are identified, like rendering 3D graphics, decoding video, or in this case TLS encryption. Not everything is destined for the kernel as userspace has generally desirable properties as well.
  - georgia_peach 5y ago
    Last I checked, accelerated crypto instructions were unprivileged. Going by the article, this is just jamming the entire TLS hokey-pokey into kernelspace to avoid a copy. TLS session management is rather hairy. Judging by their Linux numbers, I'd take the performance hit over pushing something that complicated into the kernel.
    - megous 5y ago
      There are also DMA based crypto accelerators (quite common, almost all computers in my home have one). Even those cheap $10 Orange Pi's.
    - CyberRabbi 5y ago
      To be fair the hardware in question is not just accelerated crypto operations. From what I’ve read on this topic there are network cards that handle the end-to-end TLS protocol (sans negotiation). In general you have a point but it’s a judgement call like many things in engineering. Even in the absence of specialized TLS hardware, TLS operations are so common, there is a strong case for pushing it in the kernel if that improves efficiency by a double digit percentage. SSL_sendfile in particular is an efficiency boon for large static site hosts, it could result in significantly less hardware waste and/or reduced power consumption.
      - 5y ago

156 points by limoce 5y ago

schoen 5y ago

(2021)

KarlKemp 5y ago
November of 2021, to be exact. If you want to tag this article, you will have to tag others with (2022) sooner than one might expect.
- 1500100900 5y ago
  There's so much churn on hacker news that everything gets old after a couple of nanoseconds. So I think it would be best to just add a field for a 128-bit timestamp of the original submission and present it with high precision.

rascul 5y ago

> Alpine Linux 3.11–3.14 – Kernel is built with the CONFIG_TLS=n option, which disables building kTLS as a module or as part of the kernel. I wonder if this is still the case with 3.15? Edit: I figured I could check for myself. I don't know for sure what the default kernel package is, but there apparently is a linux-lts package. After installing this package, it leaves a config-lts file in /boot which, when grepped, returns: # CONFIG_TLS is not set The more I learn about Alpine (and musl), the more I don't want to use them. It appears as if I have an inherent performance penalty serving https web sites with nginx when I do it from Alpine.

limoce 5y ago
What if I'm running NGINX inside a Alpine-based docker container, but the host OS is Ubuntu? I guess the kernel is okay but OpenSSL provided by Alpine is not ready for kTLS, so NGINX will not enable kTLS as well.
- rascul 5y ago
  I didn't consider containers. I mostly run Alpine in virtual machines.
  - rascul 5y ago
    Hah I just realized I could check the kernel config in a vm. Earlier I looked for a kernel package to install in a container to see what the config was. Maybe I've been drinking too much. Anyway, in one of my Alpine virtual machines, /boot/config-virt has CONFIG_TLS unset.
    - bigbizisverywyz 5y ago
      The article actually mentions Alpine as one of the dist. where it's not supported by default: >The following OSs do not support kTLS, for the indicated reason: >Alpine Linux 3.11–3.14 – Kernel is built with the CONFIG_TLS=n option, which disables building kTLS as a module or as part of the kernel. and even recommends building OpenSSL and Nginx 3.0 yourself anyway, so looks like it will be a while before this might be available out-of-the-box for most major dists. But of course everything is OSS so you can DIY if you don't mind getting some ./configure under your fingernails :)
Shared404 5y ago
The way I see it is that it's just trade-off's to be chosen between. I like alpine because it's simple enough even for me to wrap my head around and understand what's going on. None of what I'm serving is high traffic or complex enough for this to matter to my usecase - and I suspect this applies to many people's situation. I appreciate musl/alpine for their stability and simplicity I suppose, and a bit of performance is an OK price to pay in my mind.
stormbrew 5y ago
> It appears as if I have an inherent performance penalty serving https web sites with nginx when I do it from Alpine. This is a weirdly alarmist take on this? If you're trying to use bleeding edge kernel features, which this basically is, you should probably feel comfortable using an alternative kernel because odds are pretty good you're gonna have to update sooner rather than later for some bug fix or other. It's just not really reasonable to expect all distros to enable all kernel flags all the time, a lot of them are not really proven safe or secure. Especially when they're new.
- rascul 5y ago
  > This is a weirdly alarmist take on this It's an observation. I didn't intend for it to be alarmist, if you interpreted it that way then perhaps I could have worded it differently. > If you're trying to use bleeding edge kernel features, which this basically is It's been in the kernel since 2017 (the article noted kernel 4.13 which was released 2017 when I looked it up). That doesn't seem very bleeding edge to me. > It's just not really reasonable to expect all distros to enable all kernel flags all the time Of course. And I've already been considering moving away from Alpine for at least some use cases, and this can lead me to use move away for more use cases.
  - stormbrew 5y ago
    > It's an observation. I didn't intend for it to be alarmist, if you interpreted it that way then perhaps I could have worded it differently. Alarmist is perhaps the wrong word. What I mean is that this is a very strange and high bar for choosing a distro. You aren't "suffering a penalty by using alpine," you're being a beta tester by using a non-LTS ubuntu with a bunch of random flags on or whatever. You can also just.. use a different kernel version with alpine (or whatever distro), no one's stopping you. > It's been in the kernel since 2017 (the article noted kernel 4.13 which was released 2017 when I looked it up). That doesn't seem very bleeding edge to me. You can't actually use the version in 4.13 though, you need at least 4.17, because apparently that's what openssl 3.0.0 requires. Now 4.17 has been around for a while too! But you also need openssl 3.0.0 to make practical use of it, and that's only been out since sept 2021. And also had a massive number of breaking changes. And then you have to be using a newer kernel version than that to get tlsv3 ciphers apparently. Looks like somewhere around 5.10, though it doesn't explicitly say in the article afaict. If you don't use that then maybe you're gaining some speed but you're also downgrading your security. And then you need to use bleeding edge nginx and manually compile that against openssl3. So yeah. It's technically been there for years. But in practical terms no one (or very few people at least) have been using it in anger until the last few months. A new syscall in linux is "bleeding edge" for a while. Honestly the kernel is the least of your concerns here.
_ikke_ 5y ago
A feature request to enable it: https://gitlab.alpinelinux.org/alpine/aports/-/issues/13664 https://gitlab.alpinelinux.org/alpine/aports/-/issues/13664

winrid 5y ago

This would make Nchan even faster, neat.

WJW 5y ago
How so? According to the article this works only for serving static files. Nchan, cool as it is, has nothing to do with static files.
- winrid 4y ago
  Ah, I see, I merely skimmed. Thanks for the correction.

drewg123 5y ago

We've been running kTLS + SSL sendfile on FreeBSD at Netflix for the last 6 or 7 years. (We had local patches to nginx, before nginx did them "right", and 2 versions of kTLS before the 2nd version was upstreamed to FreeBSD). The savings in terms of CPU use and memory BW are pretty substantial. Especially when you use a NIC which can do in-line kTLS offload, then things basically go back to pre-TLS costs because the buffers are not touched at all by the CPU. BTW, FreeBSD 14 supports cha-cha poly. But is far more CPU intensive than GCM, so I'd advise against using it.

cperciva 5y ago
Random question: Do you force Netflix clients onto the ciphers which are most efficient for the Netflix servers, or are there cases (I'm thinking mobile devices particularly) where it makes sense to use the ciphers which are most efficient for the clients?
- jchw 5y ago
  Pretty sure the best choice is basically always AES GCM because most modern chipsets can hardware offload that. Curious to hear the answer, though.
  - stragies 5y ago
    Genuine quesion: In 2022, is that still dependent on "Can I trust my (HW?)random number generator"?
    - jchw 5y ago
      I don’t believe so. AES is symmetric-key encryption; if you do it incorrectly, it doesn’t decrypt at all. The only place you can mess up, then, is in the mode, which I don’t believe is accelerated by most (any?) AES encryption instruction sets; instead, they mostly handle doing an individual round of encryption/decryption. Public key crypto systems seem like they would be much scarier to have hardware acceleration, though I’m sure if you broke it down to low level enough bits you could make it impossible for the hardware to “break” it (aside from, well, if it decided to just maliciously subsitute your code for its own. But it could do that without extension ISAs.)

vsh_05 5y ago

Aissen 5y ago

Just a note to anyone wanting to use kTLS: make sure to benchmark it first, like in the article. Depending on the CPU architecture, it might even be slower than plain userspace TLS. Also, while the tx side has seen lots of investment (from CDN companies/owners), the receive side usually comes later. For instance, it's not supported for TLS 1.3 in openssl (although there's an open PR).

georgia_peach 5y ago

How long before we push everything into the kernel?

CyberRabbi 5y ago
Generally the kernel is involved when it comes to making use of hardware. Specialized hardware emerges when widely applicable bottlenecks are identified, like rendering 3D graphics, decoding video, or in this case TLS encryption. Not everything is destined for the kernel as userspace has generally desirable properties as well.
- georgia_peach 5y ago
  Last I checked, accelerated crypto instructions were unprivileged. Going by the article, this is just jamming the entire TLS hokey-pokey into kernelspace to avoid a copy. TLS session management is rather hairy. Judging by their Linux numbers, I'd take the performance hit over pushing something that complicated into the kernel.
  - megous 5y ago
    There are also DMA based crypto accelerators (quite common, almost all computers in my home have one). Even those cheap $10 Orange Pi's.
  - CyberRabbi 5y ago
    To be fair the hardware in question is not just accelerated crypto operations. From what I’ve read on this topic there are network cards that handle the end-to-end TLS protocol (sans negotiation). In general you have a point but it’s a judgement call like many things in engineering. Even in the absence of specialized TLS hardware, TLS operations are so common, there is a strong case for pushing it in the kernel if that improves efficiency by a double digit percentage. SSL_sendfile in particular is an efficiency boon for large static site hosts, it could result in significantly less hardware waste and/or reduced power consumption.
    - 5y ago