4 ms·

Let's Encrypt worked well for me the other day, except that I hit a firewall problem. I firewall all "new connection" outbound traffic, so that if a web app get

by rlpb 10y ago

Let's Encrypt worked well for me the other day, except that I hit a firewall problem. I firewall all "new connection" outbound traffic, so that if a web app gets compromised, it can't start sending spam or try to spread a compromise to other sites. This caused Let's Encrypt to fail. I worked around this by temporarily allowing outbound connections, but I wonder whether this needs to be fixed more generally? It seems reasonable to me to block all outbound connections on a web server that isn't expected to need any. But Let's Encrypt's (perfectly reasonable) need to renew certificates breaks this assumption. Unfortunately, it doesn't seem easy to set up an exception on an IP-based firewall to allow Let's Encrypt outbound connections only. This makes automated renewals hard.

mholt 10y ago
Use the DNS challenge: no open ports or listeners on your machine, just set a DNS record.
- rlpb 10y ago
  No, it's the outbound API connection that is the problem. The server listens on HTTPS from the world, so having that listening inbound wasn't a problem.
  - ygjb-dupe 10y ago
    Right, so you can run the dns01 challenge on a separate service, up to and including running it in jenkins to automatically renew and push your certs, and kick the nginx server. I should document how my setup works and put it up somewhere...
q3k 10y ago
If you're using a local iptables ruleset as your firewall, you can use the owner module [1] to filter by UID/GID, for example allow establishing connections from the user that runs your LE automation. [1] - https://www.frozentux.net/iptables-tutorial/iptables-tutorial.html#OWNERMATCH https://www.frozentux.net/iptables-tutorial/iptables-tutoria...
mikeash 10y ago
Is it possible to fix it more generally? You need to make the renewal request somehow, and that means there has to be an outbound request at some point.
- rlpb 10y ago
  > Is it possible to fix it more generally? It's a good question. I'm not sure. It is, however, a caveat of the Let's Encrypt experience for me - whether or not it is possible to fix. > You need to make the renewal request somehow, and that means there has to be an outbound request at some point. Agreed. It would be nice if there were some way to set up an (external) IP-based firewall such that the Let's Encrypt API calls can go through, but nothing else. One way might be to declare publish destination IPs as static (rather than dynamically through DNS), but of course this would be rather brittle for them from a service perspective. Another way might be to get a port number assigned for the ACME protocol, rather than overloading HTTPS. ACME services could operate on both an official ACME port number as well as the HTTPS port for backwards compatibility. The protocol could remain HTTPS-based. Then firewalls could explicitly allow ACME through, while blocking outbound HTTPS for anything else that may be an attempt to exploit a general HTTPS server rather than being known to be ACME-specific.
  - mikeash 10y ago
    How practical would it be to route the outbound requests through another machine (or VM?) which has more privileges, but is locked down to do nothing but send ACME requests? Or have that VM be entirely in charge of certificate renewal, and give it access to your certificate store somehow?
    - rlpb 10y ago
      > Or have that VM be entirely in charge of certificate renewal, and give it access to your certificate store somehow? This is what I'd prefer, I think. The thing is, part of the appeal of Let's Encrypt is the readily available ACME client that JFDI. Such an easy-to-use client doesn't currently exist for this use case.
pfg 10y ago
The common solution to this problem is to deploy a HTTP(S) proxy (like Squid) for outgoing connections. These can operate based on things like the Host header rather than the destination IP. Your web server would be locked down to only allow outgoing connections to this proxy.
ygjb-dupe 10y ago
Have a separate host that handles all of the renewals (e.g. dns01 challenge), then pushes the updated cert to your webserver and reloads the nginx (or whatever process is required for your webserver/proxy).

by rlpb 10y ago

mholt 10y ago

Use the DNS challenge: no open ports or listeners on your machine, just set a DNS record.

rlpb 10y ago
No, it's the outbound API connection that is the problem. The server listens on HTTPS from the world, so having that listening inbound wasn't a problem.
- ygjb-dupe 10y ago
  Right, so you can run the dns01 challenge on a separate service, up to and including running it in jenkins to automatically renew and push your certs, and kick the nginx server. I should document how my setup works and put it up somewhere...

q3k 10y ago

If you're using a local iptables ruleset as your firewall, you can use the owner module [1] to filter by UID/GID, for example allow establishing connections from the user that runs your LE automation. [1] - https://www.frozentux.net/iptables-tutorial/iptables-tutorial.html#OWNERMATCH https://www.frozentux.net/iptables-tutorial/iptables-tutoria...

mikeash 10y ago

Is it possible to fix it more generally? You need to make the renewal request somehow, and that means there has to be an outbound request at some point.

rlpb 10y ago
> Is it possible to fix it more generally? It's a good question. I'm not sure. It is, however, a caveat of the Let's Encrypt experience for me - whether or not it is possible to fix. > You need to make the renewal request somehow, and that means there has to be an outbound request at some point. Agreed. It would be nice if there were some way to set up an (external) IP-based firewall such that the Let's Encrypt API calls can go through, but nothing else. One way might be to declare publish destination IPs as static (rather than dynamically through DNS), but of course this would be rather brittle for them from a service perspective. Another way might be to get a port number assigned for the ACME protocol, rather than overloading HTTPS. ACME services could operate on both an official ACME port number as well as the HTTPS port for backwards compatibility. The protocol could remain HTTPS-based. Then firewalls could explicitly allow ACME through, while blocking outbound HTTPS for anything else that may be an attempt to exploit a general HTTPS server rather than being known to be ACME-specific.
- mikeash 10y ago
  How practical would it be to route the outbound requests through another machine (or VM?) which has more privileges, but is locked down to do nothing but send ACME requests? Or have that VM be entirely in charge of certificate renewal, and give it access to your certificate store somehow?
  - rlpb 10y ago
    > Or have that VM be entirely in charge of certificate renewal, and give it access to your certificate store somehow? This is what I'd prefer, I think. The thing is, part of the appeal of Let's Encrypt is the readily available ACME client that JFDI. Such an easy-to-use client doesn't currently exist for this use case.

pfg 10y ago

The common solution to this problem is to deploy a HTTP(S) proxy (like Squid) for outgoing connections. These can operate based on things like the Host header rather than the destination IP. Your web server would be locked down to only allow outgoing connections to this proxy.

ygjb-dupe 10y ago

Have a separate host that handles all of the renewals (e.g. dns01 challenge), then pushes the updated cert to your webserver and reloads the nginx (or whatever process is required for your webserver/proxy).