6 ms·
> Still, this type of oversight seems all too common even in large companies. (...) Has anyone developed a tool designed specifically to avoid certificate expir
by revvx 7y ago
> Still, this type of oversight seems all too common even in large companies. (...) Has anyone developed a tool designed specifically to avoid certificate expiry disasters?
LetsEncrypt renewal is supposed to be automated. [1]
I know of a company that hosted blogs for thousands of customers. They used LetsEncrypt, but the CTO considered automatic renewals a possible security risk, so they did it manually. Problem is, the expiration happened in a weekend and they "forgot" to update the certificates before that. Suffice to say that the next Monday wasn't pleasant. They automated after that.
[1] https://letsencrypt.org/about/ https://letsencrypt.org/about/
- mc32 7y agoSo did they conclude it wasn’t a security concern or did they conclude the security risk was worth the uptime?
- mehrdadn 7y agoI'm curious as well. My intuition would be that it's not a concern, since servers already keep their private keys stored locally in order to be able to communicate with clients anyway? Being able to update them doesn't really seem to make things any different. But I feel like I could be missing something/not have thought through it properly. (I imagine security implications can get more complicated if a different server decrypts traffic vs. processes it, etc.)
- revvx 7y agoThe "manual" process used previously by the company already involved some form of automation, so it was more about trusting CertBot not to do anything horrendous. But now that you mention it, I wonder what's the opinion of security experts like tptacek on cert renewal automation.
- inflatableDodo 7y agoWe could attempt a summoning. Quick, make a wildly inaccurate claim about the correct way to implement an encryption library.
- revvx 7y agoWhen pressed, they admitted it was just "gut feeling". The team audited a couple ACME clients and couldn't find anything to justify not automating.
- dingaling 7y agoHaving a root process with write-privileges to /etc on production machines and also able to communicate over the Internet definitely is a security risk. To mitigate that you end-up building a series of privilege-restricted jobs flowing from the DMZ back into the internal network. And maintaining that might be more complicated than just manually renewing, depending upon the processes and architecture of the company.
- Whitestrake 7y agoWhy would a process need to run as root or have write privileges to /etc in order to automate LetsEncrypt renewals? I run Caddy (which uses acme-go/lego as its ACME provider) as a non-root user with no access to /etc at all. It seems to be running fine.
- tedunangst 7y agoDepends on setup, but frequently private keys are inaccessible to the web server worker process. (Which starts as root, loads keys, drops privs, etc.)
- tialaramex 7y agoMost popular ACME (Let's Encrypt) clients allow you to provide a CSR instead of generating the keys themselves. That means a bunch more work for you, but if you're worried about this, that's what you should do. Have your safe (even manual if you insist) process make keys, make CSRs for the keys, and put those somewhere readable. The ACME client will hand them over to the CA saying "I want certs corresponding to these CSRs" without needing access to your TLS private keys at all.
- Abishek_Muthian 7y agoSome shared hosting like Bluehost now provide LetsEncrypt by default for all their sites with auto-renewal (But I don't recommend Bluehost shared plans for anything even closer to serious hobby due to absurd downtimes like most other shared hosting). I used manual renewal for LetsEncrypt for about 4 websites on other shared hosts & renewing them every 3 months was a pain; had to keep reminders and schedules just not to miss renewals until I synchronised their renewal schedules to batch (manual) renewing them. I had automated renewal for 1 website on a cloud server, it was a one time effort, I never had to bother about SSL cert for that site and the most favourable of them all.
- revvx 7y agoAnother option is using a Web Server/Reverse Proxy that supports Let's Encrypt automatically, like Caddy [1]. I believe Apache HTTPD has partial support [2], too. [1] https://caddyserver.com https://caddyserver.com [2] https://httpd.apache.org/docs/2.4/mod/mod_md.html https://httpd.apache.org/docs/2.4/mod/mod_md.html
- nsomaru 7y agoNginx works well and there's a tool that automates most of the extra config stuff for you.
- Abishek_Muthian 7y agoApache HTTPD looks interesting, so using which we renew LetsEncrypt cert without using certbot?
- revvx 7y agoIt requires some fiddling and it's in experimental state, but yes! Here's the documentation: https://github.com/icing/mod_md/wiki/Migration https://github.com/icing/mod_md/wiki/Migration
- glitchcrab 7y agoTraefik is another option here for a reverse proxy with automated renewals; I use it in a ton of places. https://traefik.io https://traefik.io
- tedunangst 7y agoI have no idea why you'd deliberately wait the full 90 days to do a manual renew. For reasons, I renew manually, but every 60 days or so. Nowhere close to the deadline.
- revvx 7y agoExactly. Their FAQ [1] recommends exactly that: renewing every 60 days. [1] https://letsencrypt.org/docs/faq/#what-is-the-lifetime-for-let-s-encrypt-certificates-for-how-long-are-they-valid https://letsencrypt.org/docs/faq/#what-is-the-lifetime-for-l...
- n42 7y agoJust curious, are you talking about Webflow? Because I had to hunt down and make sure our Let's Encrypt auto renewal was working until I realized the certificate was served by them. They wait until the last 12 hours to renew the certificate. I have no idea what type of rationalization would lead to that decision.
- tass 7y ago90 days is 4 times a year. 60 is 6 times, 50% more expensive when you’re paying someone to perform the task.
- n42 7y agoI had the same thought, but I still find that absurd. Say they host 500,000 websites with HTTPS. 1,000,000 renewals they save spread across the year, roughly 2 renewals a minute. That is pennies. A t2.medium could handle that type of load increase
- albru123 7y agoA bit OT, but what's up with this usage of Amazon EC2 tiers as a unit of computational power?
- revvx 7y agoNope, content marketing company
- owaislone 7y agoThey didn't have renew automatically but they could automate notifications, alerts or even banners in their internal apps when 60-70% of the time was exhausted. If I was given such a restriction, I'd still automate it 100% but require a human to authorize it every time by clicking a magic link in their email, slack or some dashboard, and nag them with notifications until someone authorized it.
- pmontra 7y agoIt's automated but things can go wrong even when correctly configured and tested. Real world example: certbot version got old, the renewal server didn't support it anymore, the certificate didn't renew, the web site got the dreaded https warning page. Of course that is also a kind of misconfiguration. The site has Debian security auto updates on but certbot is not among them. It should be forced to be updated. Furthermore there was no monitoring of errors in its log file. Still it's not as simple as one believes Letsencrypt to be.
- magicalhippo 7y agoThen the automatic update process stops for some reason and your certificate expires... At the end of the day, someone needs to verify that new certificates gets acquired and installed before the old ones expire. Automation makes acquiring them less tedious, but not much for making sure someone pays attention.
- bluejay2387 7y agoWe update automatically AND manually check periodically to make sure the update took place. That company must be overly fond of drama...
- djhaskin987 7y agoDidn't Mozilla invent Let's encrypt? That would make this disaster doubly embarrassing.