5 ms·

can someone please share how they deploy/distribute Let's encrypt certificates with auto renewal on load balanced multiple EC2 servers for the same dns name. I

can someone please share how they deploy/distribute Let's encrypt certificates with auto renewal on load balanced multiple EC2 servers for the same dns name. I had tried this a while but had to give up and just bought SSL certs which I then include in my EC2 image.

GuyPostington 7y ago
You can store the cert and key in a key/val store and each EC2 server would only need to renew the cert.
aloknnikhil 7y ago
Have you tried using Caddy? It handles automatic HTTPS (a.k.a Let's Encrypt) renewal across a fleet. https://caddyserver.com/docs/automatic-https#fleet https://caddyserver.com/docs/automatic-https#fleet
- linsomniac 7y ago
  I have, I set it up around 6 months ago on a couple of my own personal sites, and it seems to work pretty well. It took a while to figure out the right way to configure it for multiple site hosting, but it has been pretty trouble-free. The cert registration and renewal was really slick. If you want to see it "in action" on a single EC2 instance: https://pythoneers.org/ https://pythoneers.org/ I used an Ansible role to provision it, antonier77/caddy-ansible and it has worked nicely. But, as another comment mentioned: If you are in an AWS load balancer, you probably want to use the AWS certificates.
- manishsharan 7y ago
  Caddy seems nice but I am way too invested in Nginx
  - mholt 7y ago
    This reminds me... we have an nginx config adapter started. Would you be willing to help try it when we get it closer to being done? It will let you bring your nginx config and converts it to a Caddy config.
markstos 7y ago
If your load balancers are AWS load balancers, you are doing it wrong. AWS provides their own free certificates that work the load balancer and they handle renewal as well. Let's Encrypt would be an unnecessary additional dependency and complexity. (At least if you are comfortable terminating your SSL at the load balancer. If you plan to use SSL between the AWS LBs and your EC2 instances, then AWS certs don't work there and you'll need to provision them yourself using something like LE).
- owenmarshall 7y ago
  > (At least if you are comfortable terminating your SSL at the load balancer. If you plan to use SSL between the AWS LBs and your EC2 instances, then AWS certs don't work there and you'll need to provision them yourself using something like LE). Interesting note - ALBs/ELBs (NLBs with SSL termination as well, I would assume, but I am not sure) do not perform validation of your backend certificate. You can terminate at the load balancer and use an expired self signed SHA1 cert for all AWS cares.
  - abofh 7y ago
    Elbs do support public key verification of the backends (search for Enable backend authentication). I believe you are correct wrt Albs and nlbs, and in neither case does it check the cert ttbomlk, just the public key.
- manishsharan 7y ago
  I stopped using AWS load balancer as I found that it was costing me more than my server, which are cheap as I normally use a fleet of spot instances . Also this is not corporate funded but my own money -- so I am very cost sensitive
notyourday 7y ago
I pick up the URL for validation request at the edges and pass it to a specialized backend that handles validation. Certs/keys get added to a key:value store that is monitored by the edges. Each edge know the timestamp of the current key:cert pair that the edge successfully wrote (if there was an update) or the timestamp of the current key:cert pair that the config-baker wrote during the edge built process if edge has never wrote a key key:cert pair. If the timestamp is newer, then the edge updates the key/cert and restarts itself with a new cert. The key:value store for keys is also monitored by a config-baker. When a config-baker detects a change in key pairs, it writes a new initial configuration JSON with new keys/certs which is stored in the infrastructure management git. So when a new edge is built and launched by the infrastructure policy enforcer, it would immediately have the keys on it as soon as it comes into service at which point it will become just another edge following the same "protocol" Edit: Currently it manages 671 certificates on 16 different edges. After a new key:cert is published in a "go" mode it rolls out in ~1 minute to all the edges.
linsomniac 7y ago
For our dev/stg environment we use certbot, and haproxy has a backend configured to pass the validation requests back to certbot based on "/.well-known/acme-challenge" in the path. For production we are only using it, currently, for the cert at the backup location, and we couldn't use certbot because of the way it is packaged for Ubuntu, it wouldn't work with Route53 DNS validation. Because it's the backup site, HTTP requests aren't normally directed at the server doing the requests. So I switched to "acme.sh" and that's been really reliable.
chc 7y ago
We use Terraform to automate the actual generation of the certificates (it checks the expiration date and does a renewal request when you're 30 days away), but the basic approach we use is to generate the certificate with DNS validation, store it in a known location in S3, and then have the machines download the certificate on startup and refresh as a cronjob.
doliveira 7y ago
I don't know if that is a good practice or whatnot, but I just set up .well-known in my nginx config to redirect to an S3 bucket where I write the ACME challenge to.

GuyPostington 7y ago

You can store the cert and key in a key/val store and each EC2 server would only need to renew the cert.

aloknnikhil 7y ago

Have you tried using Caddy? It handles automatic HTTPS (a.k.a Let's Encrypt) renewal across a fleet. https://caddyserver.com/docs/automatic-https#fleet https://caddyserver.com/docs/automatic-https#fleet

linsomniac 7y ago
I have, I set it up around 6 months ago on a couple of my own personal sites, and it seems to work pretty well. It took a while to figure out the right way to configure it for multiple site hosting, but it has been pretty trouble-free. The cert registration and renewal was really slick. If you want to see it "in action" on a single EC2 instance: https://pythoneers.org/ https://pythoneers.org/ I used an Ansible role to provision it, antonier77/caddy-ansible and it has worked nicely. But, as another comment mentioned: If you are in an AWS load balancer, you probably want to use the AWS certificates.
manishsharan 7y ago
Caddy seems nice but I am way too invested in Nginx
- mholt 7y ago
  This reminds me... we have an nginx config adapter started. Would you be willing to help try it when we get it closer to being done? It will let you bring your nginx config and converts it to a Caddy config.

markstos 7y ago

If your load balancers are AWS load balancers, you are doing it wrong. AWS provides their own free certificates that work the load balancer and they handle renewal as well. Let's Encrypt would be an unnecessary additional dependency and complexity. (At least if you are comfortable terminating your SSL at the load balancer. If you plan to use SSL between the AWS LBs and your EC2 instances, then AWS certs don't work there and you'll need to provision them yourself using something like LE).

owenmarshall 7y ago
> (At least if you are comfortable terminating your SSL at the load balancer. If you plan to use SSL between the AWS LBs and your EC2 instances, then AWS certs don't work there and you'll need to provision them yourself using something like LE). Interesting note - ALBs/ELBs (NLBs with SSL termination as well, I would assume, but I am not sure) do not perform validation of your backend certificate. You can terminate at the load balancer and use an expired self signed SHA1 cert for all AWS cares.
- abofh 7y ago
  Elbs do support public key verification of the backends (search for Enable backend authentication). I believe you are correct wrt Albs and nlbs, and in neither case does it check the cert ttbomlk, just the public key.
manishsharan 7y ago
I stopped using AWS load balancer as I found that it was costing me more than my server, which are cheap as I normally use a fleet of spot instances . Also this is not corporate funded but my own money -- so I am very cost sensitive

notyourday 7y ago

I pick up the URL for validation request at the edges and pass it to a specialized backend that handles validation. Certs/keys get added to a key:value store that is monitored by the edges. Each edge know the timestamp of the current key:cert pair that the edge successfully wrote (if there was an update) or the timestamp of the current key:cert pair that the config-baker wrote during the edge built process if edge has never wrote a key key:cert pair. If the timestamp is newer, then the edge updates the key/cert and restarts itself with a new cert. The key:value store for keys is also monitored by a config-baker. When a config-baker detects a change in key pairs, it writes a new initial configuration JSON with new keys/certs which is stored in the infrastructure management git. So when a new edge is built and launched by the infrastructure policy enforcer, it would immediately have the keys on it as soon as it comes into service at which point it will become just another edge following the same "protocol" Edit: Currently it manages 671 certificates on 16 different edges. After a new key:cert is published in a "go" mode it rolls out in ~1 minute to all the edges.

linsomniac 7y ago

For our dev/stg environment we use certbot, and haproxy has a backend configured to pass the validation requests back to certbot based on "/.well-known/acme-challenge" in the path. For production we are only using it, currently, for the cert at the backup location, and we couldn't use certbot because of the way it is packaged for Ubuntu, it wouldn't work with Route53 DNS validation. Because it's the backup site, HTTP requests aren't normally directed at the server doing the requests. So I switched to "acme.sh" and that's been really reliable.

chc 7y ago

We use Terraform to automate the actual generation of the certificates (it checks the expiration date and does a renewal request when you're 30 days away), but the basic approach we use is to generate the certificate with DNS validation, store it in a known location in S3, and then have the machines download the certificate on startup and refresh as a cronjob.

doliveira 7y ago

I don't know if that is a good practice or whatnot, but I just set up .well-known in my nginx config to redirect to an S3 bucket where I write the ACME challenge to.