6 ms·

On the fly SSL registration and renewal inside Nginx with Let's Encrypt

234 points by snaky 10y ago

msumpter 10y ago
Awesome! I had been playing around with https://github.com/jwilder/nginx-proxy https://github.com/jwilder/nginx-proxy and docker-gen to handle automatic Let's Encrypt generation for my docker containers. It's amazing how quick to get some of these thing setup with very work: https://github.com/msumpter/docker-lua-resty-auto-ssl https://github.com/msumpter/docker-lua-resty-auto-ssl
koolba 10y ago
Hey this is pretty cool. Would be even cooler if it held off on the SSL handshake with the initial client until after the certificate was issued by LetsEncrypt. If it's only a couple of seconds it might be possible.
- aavotins 10y ago
  Or it would be even greater if it requested/generated all certificates upon configuration file parsing and delayed nginx from taking those values into account until valid certificates have been obtained.
  - nickblah 10y ago
    Author here. Our current approach is perhaps a bit different, since we're not actually parsing the nginx config file, so we don't have knowledge of the domains at startup. Instead, we're relying on the "allow_domain" Lua function to be defined which provides a way to determine which domains should be allowed. By making this a Lua function, it allows for the logic behind this to be very flexible and dynamic (for example, nginx could handle wildcard requests to any domain, and then you could lookup what domains to allow SSL registration for from another dynamic source you might already have). But I do like the idea of allowing this to be handled at startup too. Thanks for the idea!
- nickblah 10y ago
  Author here.. If I'm understanding you correctly, I believe that's how things are already working. The very first time a client hits a new domain, the SSL handshake initiates the certificate registration with Let's Encrypt (assuming the domain is part of the whitelist of allowed domains). The certificate is registered in the background, while this first request is paused. The SSL handshake is then completed with the first client once the certificate is successfully issued. This does lead to the very first client's request being delayed a few seconds, but this is a one-time delay (per allowed domain).
  - gonzo 10y ago
    Nice work! We'll look into including this in pfsense 2.4.
  - koolba 10y ago
    Ah I didn't realize that. I figured the first request kicks off the request process so it's available for subsequent requests. This is even cooler!
eloy 10y ago
Please, please stop using the name "SSL" and use "TLS". As others replied, x509 would be even more correct, but SSL is completely incorrect, because that protocol is insecure and shouldn't be used. Using SSL suggest that the software supports that protocol, although it doesn't (as it should)
- everfree 10y ago
  If we're getting nitpicky, let's use the term TLS to describe the protocol and X.509 to describe the certificates and renewals.
- blowski 10y ago
  As Shakespeare might once have said: What's in a name? That which we call TLS By any other name would still offer encryption.
artursapek 10y ago
Wow, this is awesome. LetsEncrypt is great but I still find it somewhat clumsy to use it in production. This is the first no-downtime solution I've seen. The certificate cartel is slowly dying :)
- snaky 10y ago
  You can even update Nginx binary with no downtime - https://nginx.org/en/docs/control.html#upgrade https://nginx.org/en/docs/control.html#upgrade
bmurphy1976 10y ago
Apparently this requires OpenResty. OpenResty is an app server type platform built on Nginx. This does not appear to work on your standard run-of-the-mill Nginx system package. OpenResty is a cool project, but it brings a lot of additional infrastructure along with it that you may not want. It would be nice of this was clearly indicated in the title of this post. Updated: see comment below, the README has been clarified, you can run this with the nginx_lua extension. You don't necessarily need all of OpenResty.
- snaky 10y ago
  Actually it does not require OpenResty, just ngx_lua module.
  - dsp1234 10y ago
    Someone should update the Installation section. It currently states: Requirements: OpenResty 1.9.7.2 or higher
    - nickblah 10y ago
      I've updated the requirements in the README. Thanks for pointing that out!
      - bmurphy1976 10y ago
        OK, that's great! I recant my earlier comment. :)
        snaky 10y ago
        btw, what is a 'lot of additional infrastructure' that OpenResty brings?
- gtirloni 10y ago
  Compiling nginx from scratch to include ngx_lua is a bummer because of all the added questions that need to be answered (packaging, repositories, automation, etc). It becomes 98% of the time spent enabling this awesome feature (at least on CentOS and Fedora, from what I can tell there aren't RPM packages for ngx_lua).
LoSboccacc 10y ago
ah this would nicely handle the let's encrypt wildcard issue. too bad I just forked $$$ for a wildcard cert. but hey will turn off renew and have a go with this :)
- schoen 10y ago
  Bear in mind that you could potentially hit a rate limit depending on the number of subdomains you have (or a different rate limit depending on how the plugin requests particular certs). I'm sure a small number of users will become very aware of this issue very soon. This is a reason that being at least somewhat aware of the overall list of names to be covered ahead of time could be helpful.
wyattjoh 10y ago
Not having this featured built into NGINX is the reason why I transitioned my production servers over from it to Caddy. Having SSL just "dealt with" is the biggest time saver ever.
kodablah 10y ago
Nice. Good to see this has a shared storage option (Redis) for load balanced servers. The LE HTTP challenge gives no guarantee which A record it will use. Similarly, if your LB chooses another server even behind the same IP, it'd have to share the challenge state. I haven't read deeply, but I assume it handles the race conditions that can occur with auto renewal (i.e. different servers triggering auto renewal at the same time, overusing LE and hitting the quota faster). It was something I struggled with during the implementation of pluggable storage adapters for Caddy[0]. 0 - https://github.com/mholt/caddy/pull/913 https://github.com/mholt/caddy/pull/913
- andrewstuart2 10y ago
  Re: hitting the quota faster, they have a staging environment that they explicitly mention as a target for client development. Probably not a helpful tip to you now, but others may find it useful. https://letsencrypt.org/docs/staging-environment/ https://letsencrypt.org/docs/staging-environment/
- okket 10y ago
  If you are going this scale, there is another way: Use your own DNS server and try the DNS-01 challenge. My setup works with [0]letsencrypt.sh, [1]pdns_api.sh, [2]PowerDNS4 (MySQL backend w/ replication) and a little script that distributes the certs and restarts services. The nice thing is, this works for anything you can put in DNS, you do not need not to respond to a HTTP request for a name (think mail server). [0] https://github.com/lukas2511/letsencrypt.sh https://github.com/lukas2511/letsencrypt.sh [1] https://github.com/silkeh/pdns_api.sh https://github.com/silkeh/pdns_api.sh [2] https://github.com/PowerDNS/pdns https://github.com/PowerDNS/pdns
- nickblah 10y ago
  Yes, as long as you're using a shared storage adapter (like Redis), then for the most part, duplicate registration or renewal requests should be handled. The approach to locking we're using is somewhat simplistic (and the specifics depend on the storage adapter you're using), so there might be some rare edge-cases in which 2 requests to Let's Encrypt slip through. However, that shouldn't actually affect the functionality (the last response simply wins). Here's a bit more detail in the code: https://github.com/GUI/lua-resty-auto-ssl/blob/v0.8.2/lib/resty/auto-ssl/storage.lua#L73-L83 https://github.com/GUI/lua-resty-auto-ssl/blob/v0.8.2/lib/re...
tinganho 10y ago
Just did something similar. Though with HAProxy inside a Docker container. https://github.com/tinganho/haproxy-with-letsencrypt-auto-renewal https://github.com/tinganho/haproxy-with-letsencrypt-auto-re...
finnn 10y ago
This is really cool. Have you considered implimenting the DVSNI challenge?
- nickblah 10y ago
  Author here. I hadn't really considered it, but it should be possible. Since we're handling this with nginx and Lua, it made it pretty trivial to handle the simple HTTP challenge (since we can easily intercept the /.well-known/acme-challenge request), so that's why we went with that approach, but other approaches should be possible.
  - finnn 10y ago
    I guess I figured DVSNI should be even easier since you're already evaluating the domains of incoming TLS connections and picking which cert to respond with from there.
    - nickblah 10y ago
      I haven't actually looked into DVSNI in too much detail before, so that could definitely be the case. I'll have to investigate a bit more (or any pull requests are always welcome). Thank you for the tip!
      - finnn 10y ago
        No problem, thanks for building this!
sandGorgon 10y ago
This is awesome.. hopefully we can bake nginx inside docker. Without something like this, we could not spin up nginx because there was a circular dependency on the certificate files.
- hobs 10y ago
  I used a combo of https://github.com/dmp1ce/nginx-proxy-letsencrypt https://github.com/dmp1ce/nginx-proxy-letsencrypt and https://github.com/jwilder/nginx-proxy https://github.com/jwilder/nginx-proxy with no problems besides normal nginx configuration. Does assume you will have a shared volume so you can save/check certs, but it works great for my use case.
  - sandGorgon 10y ago
    Well the shared volume to "seed" a certificate is a no go for us. I would rather spend 6 bucks for a cert.
    - hobs 10y ago
      Totally get it, just thought I would share.
njharman 10y ago
Let's Encrypt is awesome. I've used it to setup real https on several of my domains. It was easy, automatically renews, just works.
vizza 10y ago
To make sure you get notified when your certs expire, use https://letsmonitor.org https://letsmonitor.org
deleted 10y ago
[deleted]
ssahoo 10y ago
So taking extra lua dependency burden, and risk just to save 50 bucks to buy a real SSL certificate? I'm going to call it certificateless SSL.
d33 10y ago
Is there anything that would prevent an attack involving bogus SNI requests in order to trigger a DOS?
- nickblah 10y ago
  Yes, by default the module requires that you explicitly whitelist the domains you want to allow certificate registration for. So while you could allow any domain to be registered, that's not recommended for this precise reason. But the whitelist is defined as a Lua function, so you have quite a bit of flexibility in integrating the whitelist logic with other sources of information. See https://github.com/GUI/lua-resty-auto-ssl#precautions https://github.com/GUI/lua-resty-auto-ssl#precautions and the "allowed_domain" configuration for a bit more detail.
bsamuels 10y ago
I wonder if there would be a way to integrate Let's Encrypt with router admin interfaces. It would be really cool if consumer routers used a valid https certificate on their web ui
noeleon 10y ago
The shared storage redis backed option is great. I hacked together something that served .well-known from an NFS mount to overcome my issues when behind a proxy / LB. My $work has a very terribe DNS infrastructure that needs lots of work to implement dns-01.
SixSigma 10y ago
You can use StartSSL free certs for TLS too https://startssl.com/ https://startssl.com/ I set up Courier today to use it for TLS imap http://maht0x0r.blogspot.co.uk/2016/07/another-server-more-courier-fun.html http://maht0x0r.blogspot.co.uk/2016/07/another-server-more-c...

234 points by snaky 10y ago

Awesome! I had been playing around with https://github.com/jwilder/nginx-proxy https://github.com/jwilder/nginx-proxy and docker-gen to handle automatic Let's Encrypt generation for my docker containers. It's amazing how quick to get some of these thing setup with very work: https://github.com/msumpter/docker-lua-resty-auto-ssl https://github.com/msumpter/docker-lua-resty-auto-ssl

koolba 10y ago

Hey this is pretty cool. Would be even cooler if it held off on the SSL handshake with the initial client until after the certificate was issued by LetsEncrypt. If it's only a couple of seconds it might be possible.

aavotins 10y ago
Or it would be even greater if it requested/generated all certificates upon configuration file parsing and delayed nginx from taking those values into account until valid certificates have been obtained.
- nickblah 10y ago
  Author here. Our current approach is perhaps a bit different, since we're not actually parsing the nginx config file, so we don't have knowledge of the domains at startup. Instead, we're relying on the "allow_domain" Lua function to be defined which provides a way to determine which domains should be allowed. By making this a Lua function, it allows for the logic behind this to be very flexible and dynamic (for example, nginx could handle wildcard requests to any domain, and then you could lookup what domains to allow SSL registration for from another dynamic source you might already have). But I do like the idea of allowing this to be handled at startup too. Thanks for the idea!
nickblah 10y ago
Author here.. If I'm understanding you correctly, I believe that's how things are already working. The very first time a client hits a new domain, the SSL handshake initiates the certificate registration with Let's Encrypt (assuming the domain is part of the whitelist of allowed domains). The certificate is registered in the background, while this first request is paused. The SSL handshake is then completed with the first client once the certificate is successfully issued. This does lead to the very first client's request being delayed a few seconds, but this is a one-time delay (per allowed domain).
- gonzo 10y ago
  Nice work! We'll look into including this in pfsense 2.4.
- koolba 10y ago
  Ah I didn't realize that. I figured the first request kicks off the request process so it's available for subsequent requests. This is even cooler!

eloy 10y ago

Please, please stop using the name "SSL" and use "TLS". As others replied, x509 would be even more correct, but SSL is completely incorrect, because that protocol is insecure and shouldn't be used. Using SSL suggest that the software supports that protocol, although it doesn't (as it should)

everfree 10y ago
If we're getting nitpicky, let's use the term TLS to describe the protocol and X.509 to describe the certificates and renewals.
blowski 10y ago
As Shakespeare might once have said: What's in a name? That which we call TLS By any other name would still offer encryption.

artursapek 10y ago

Wow, this is awesome. LetsEncrypt is great but I still find it somewhat clumsy to use it in production. This is the first no-downtime solution I've seen. The certificate cartel is slowly dying :)

snaky 10y ago
You can even update Nginx binary with no downtime - https://nginx.org/en/docs/control.html#upgrade https://nginx.org/en/docs/control.html#upgrade

bmurphy1976 10y ago

Apparently this requires OpenResty. OpenResty is an app server type platform built on Nginx. This does not appear to work on your standard run-of-the-mill Nginx system package. OpenResty is a cool project, but it brings a lot of additional infrastructure along with it that you may not want. It would be nice of this was clearly indicated in the title of this post. Updated: see comment below, the README has been clarified, you can run this with the nginx_lua extension. You don't necessarily need all of OpenResty.

snaky 10y ago
Actually it does not require OpenResty, just ngx_lua module.
- dsp1234 10y ago
  Someone should update the Installation section. It currently states: Requirements: OpenResty 1.9.7.2 or higher
  - nickblah 10y ago
    I've updated the requirements in the README. Thanks for pointing that out!
    - bmurphy1976 10y ago
      OK, that's great! I recant my earlier comment. :)
      - snaky 10y ago
        btw, what is a 'lot of additional infrastructure' that OpenResty brings?
gtirloni 10y ago
Compiling nginx from scratch to include ngx_lua is a bummer because of all the added questions that need to be answered (packaging, repositories, automation, etc). It becomes 98% of the time spent enabling this awesome feature (at least on CentOS and Fedora, from what I can tell there aren't RPM packages for ngx_lua).

LoSboccacc 10y ago

ah this would nicely handle the let's encrypt wildcard issue. too bad I just forked $$$ for a wildcard cert. but hey will turn off renew and have a go with this :)

schoen 10y ago
Bear in mind that you could potentially hit a rate limit depending on the number of subdomains you have (or a different rate limit depending on how the plugin requests particular certs). I'm sure a small number of users will become very aware of this issue very soon. This is a reason that being at least somewhat aware of the overall list of names to be covered ahead of time could be helpful.

wyattjoh 10y ago

Not having this featured built into NGINX is the reason why I transitioned my production servers over from it to Caddy. Having SSL just "dealt with" is the biggest time saver ever.

kodablah 10y ago

Nice. Good to see this has a shared storage option (Redis) for load balanced servers. The LE HTTP challenge gives no guarantee which A record it will use. Similarly, if your LB chooses another server even behind the same IP, it'd have to share the challenge state. I haven't read deeply, but I assume it handles the race conditions that can occur with auto renewal (i.e. different servers triggering auto renewal at the same time, overusing LE and hitting the quota faster). It was something I struggled with during the implementation of pluggable storage adapters for Caddy[0]. 0 - https://github.com/mholt/caddy/pull/913 https://github.com/mholt/caddy/pull/913

andrewstuart2 10y ago
Re: hitting the quota faster, they have a staging environment that they explicitly mention as a target for client development. Probably not a helpful tip to you now, but others may find it useful. https://letsencrypt.org/docs/staging-environment/ https://letsencrypt.org/docs/staging-environment/
okket 10y ago
If you are going this scale, there is another way: Use your own DNS server and try the DNS-01 challenge. My setup works with [0]letsencrypt.sh, [1]pdns_api.sh, [2]PowerDNS4 (MySQL backend w/ replication) and a little script that distributes the certs and restarts services. The nice thing is, this works for anything you can put in DNS, you do not need not to respond to a HTTP request for a name (think mail server). [0] https://github.com/lukas2511/letsencrypt.sh https://github.com/lukas2511/letsencrypt.sh [1] https://github.com/silkeh/pdns_api.sh https://github.com/silkeh/pdns_api.sh [2] https://github.com/PowerDNS/pdns https://github.com/PowerDNS/pdns
nickblah 10y ago
Yes, as long as you're using a shared storage adapter (like Redis), then for the most part, duplicate registration or renewal requests should be handled. The approach to locking we're using is somewhat simplistic (and the specifics depend on the storage adapter you're using), so there might be some rare edge-cases in which 2 requests to Let's Encrypt slip through. However, that shouldn't actually affect the functionality (the last response simply wins). Here's a bit more detail in the code: https://github.com/GUI/lua-resty-auto-ssl/blob/v0.8.2/lib/resty/auto-ssl/storage.lua#L73-L83 https://github.com/GUI/lua-resty-auto-ssl/blob/v0.8.2/lib/re...

tinganho 10y ago

Just did something similar. Though with HAProxy inside a Docker container. https://github.com/tinganho/haproxy-with-letsencrypt-auto-renewal https://github.com/tinganho/haproxy-with-letsencrypt-auto-re...

finnn 10y ago

This is really cool. Have you considered implimenting the DVSNI challenge?

nickblah 10y ago
Author here. I hadn't really considered it, but it should be possible. Since we're handling this with nginx and Lua, it made it pretty trivial to handle the simple HTTP challenge (since we can easily intercept the /.well-known/acme-challenge request), so that's why we went with that approach, but other approaches should be possible.
- finnn 10y ago
  I guess I figured DVSNI should be even easier since you're already evaluating the domains of incoming TLS connections and picking which cert to respond with from there.
  - nickblah 10y ago
    I haven't actually looked into DVSNI in too much detail before, so that could definitely be the case. I'll have to investigate a bit more (or any pull requests are always welcome). Thank you for the tip!
    - finnn 10y ago
      No problem, thanks for building this!

sandGorgon 10y ago

This is awesome.. hopefully we can bake nginx inside docker. Without something like this, we could not spin up nginx because there was a circular dependency on the certificate files.

hobs 10y ago
I used a combo of https://github.com/dmp1ce/nginx-proxy-letsencrypt https://github.com/dmp1ce/nginx-proxy-letsencrypt and https://github.com/jwilder/nginx-proxy https://github.com/jwilder/nginx-proxy with no problems besides normal nginx configuration. Does assume you will have a shared volume so you can save/check certs, but it works great for my use case.
- sandGorgon 10y ago
  Well the shared volume to "seed" a certificate is a no go for us. I would rather spend 6 bucks for a cert.
  - hobs 10y ago
    Totally get it, just thought I would share.

njharman 10y ago

Let's Encrypt is awesome. I've used it to setup real https on several of my domains. It was easy, automatically renews, just works.

vizza 10y ago

To make sure you get notified when your certs expire, use https://letsmonitor.org https://letsmonitor.org

deleted 10y ago

[deleted]

ssahoo 10y ago

So taking extra lua dependency burden, and risk just to save 50 bucks to buy a real SSL certificate? I'm going to call it certificateless SSL.

d33 10y ago

Is there anything that would prevent an attack involving bogus SNI requests in order to trigger a DOS?

nickblah 10y ago
Yes, by default the module requires that you explicitly whitelist the domains you want to allow certificate registration for. So while you could allow any domain to be registered, that's not recommended for this precise reason. But the whitelist is defined as a Lua function, so you have quite a bit of flexibility in integrating the whitelist logic with other sources of information. See https://github.com/GUI/lua-resty-auto-ssl#precautions https://github.com/GUI/lua-resty-auto-ssl#precautions and the "allowed_domain" configuration for a bit more detail.

bsamuels 10y ago

I wonder if there would be a way to integrate Let's Encrypt with router admin interfaces. It would be really cool if consumer routers used a valid https certificate on their web ui

noeleon 10y ago

The shared storage redis backed option is great. I hacked together something that served .well-known from an NFS mount to overcome my issues when behind a proxy / LB. My $work has a very terribe DNS infrastructure that needs lots of work to implement dns-01.

SixSigma 10y ago

You can use StartSSL free certs for TLS too https://startssl.com/ https://startssl.com/ I set up Courier today to use it for TLS imap http://maht0x0r.blogspot.co.uk/2016/07/another-server-more-courier-fun.html http://maht0x0r.blogspot.co.uk/2016/07/another-server-more-c...