9 ms·
LetsEncrypt Outage
- bethekidyouwant 1y agoI hope no one was migrating infra EOD West Coast
- jaeh 1y agocan't be it, it's not friday.
- rob_c 1y agoDid the LLM delete this as well?
- burnte 1y agoEither that or someone took Ambien last night, that seems to make people do crazy mistakes. ;)
- 88j88 1y agoThe response he received had a correction to the code that the user did not expect.
- rob_c 1y agoapparently don't insult the golden goose of llms or the company that gives most of it's products away for free :P
- colinbartlett 1y agoWe're seeing a lot of downstream effects of this at StatusGator. Of course any provider that relies on LetsEncrypt to issue certs (such as Heroku) is affected. One notable exception is Cloudflare: They famously no longer rely solely on LetsEncrypt.
- keysdev 1y agoShall we have some way of freely encrypting the web that is relying on one authority? Especially something that needed to be renewed every 90 or is it 40 days now. How about issuing 100 years certificates as a default?
- kyrra 1y agoMany of the cloud providers give free certs via acme. https://cloud.google.com/certificate-manager/docs/public-ca-tutorial https://cloud.google.com/certificate-manager/docs/public-ca-... (EDIT: Google is their own CA, with https://pki.goog/ https://pki.goog/ ) The browsers and security people have been pushing towards shorter certs, not longer ones. Knowing how to rotate a cert every year, if not shorter, helps when your certificate or any of your parent certs are compromised and require an emergency rotation.
- CGamesPlay 1y agoDoes AWS provide something similar? I found ACM "exportable certificates", but that involves AWS managing your private key.
- schoen 1y agoLast I knew, AWS would issue a free certificate to people using certain AWS services, but, as you say, only if Amazon is managing the private key. You can also use ACM APIs to import keys and certificates from other CAs.
- XorNot 1y agoYou've always been able to do this. Whether its useful to your clients has always been the problem. In a practical sense you likely wouldn't like the alternatives, because for most people's usage of the internet there's exactly one authority which matters: the local government, and it's legal system - i.e. most of my necessary use of TLS is for ecommerce. Which means the ultimate authority is "are you a trusted business entity in the local jurisdiction?" Very few people would have any reason to ever expand the definition beyond this, and less would have the knowledge to do so safely even if we provided the interfaces - i.e. no one knows what safety numbers in Signal mean, if I can even get them to use Signal.
- jasonthorsness 1y agoMostly this should be a non-event due to renewal long before expiration? Although huge deal I suppose for services that require issuing new certifications constantly; Let's Encrypt would be major failure mode for them. As they move to shorter-lifetime certs (6 days now https://letsencrypt.org/2025/01/16/6-day-and-ip-certs/?utm_source=chatgpt.com https://letsencrypt.org/2025/01/16/6-day-and-ip-certs/?utm_s...) this puts it in the realm of possibility that an incident could impact long-running services.
- kenshaw 1y agoI encountered this while trying to issue a new certificate for a service. As a temporary fix, started using ZeroSSL which conveniently also supports the ACME protocol. While not a big problem, if you have something like `cert-manager` being used on Kubernetes, then it requires quite a bit of reconfiguration, and you may spend a couple hours trying to figure out why a certificate hasn't been issued yet. That said, I'm unbelievably grateful for the great product (and work!) LetsEncrypt has provided for free. Hope they're able to get their infrastructure back up soon.
- jasonthorsness 1y agoLet's Encrypt was a huge deal right from the beginning. They truly moved the web forward. Here is the HN announcement: https://news.ycombinator.com/item?id=8624160 https://news.ycombinator.com/item?id=8624160 Announcement "animated" https://hn.unlurker.com/replay?item=8624160 https://hn.unlurker.com/replay?item=8624160
- VenturingVole 1y agoTruly was a radical advancement. Makes me wonder, a decade from now what will it be that we look back upon with a similar perspective?
- deleted 1y ago[deleted]
- pepa65 1y ago
- phillipseamore 1y agoI want DANE!
- rocqua 1y agoThat ship has sailed. DNSsec is not liked even a little bit. Given that control over DNS is how domain validated certs are handed out, it would make a lot of sense to cut out the middle man. But DNS does not have a good reliable authenticated transport mechanism. I wonder if there was a way to build this that would have worked.
- phillipseamore 1y agoMy biggest problem is how centralized issuance is. Half the year I live on an island that is reliant on submarine cables and has historically had weeks and months long outages and with a changing world I suspect that might become reality once again. Locally this wasn't much of an issue, the ccTLD continues to function, most services (but now about 35%) are locally hosted. Then HTTPS comes along. Zero certificates could be (re-)issued during an outage. A locally run CA isn't really an option (standalone simply isn't feasible and getting into root stores takes time and money), so you are left with teaching users to ignore certificate errors a few weeks into an extended outage. I could see someone like LE working with TLD registrars to enable local issuance (with delegated/sub-CA certificates restricted to the TLD), that could also mitigate problems like today (decentralize issuance) and the registrars are already the primary source of truth for DV validation.
- ocdtrekkie 1y agoRealistically there's no reason except Google retaining centralized control of the Internet for there to be a specific group of trusted CAs that meet Google's arcane specifications which can issue certificates the entire world trusts. Your registrar should be able to validate your ownership of the domain, ergo your registrar should be your CA. Instead of a bunch of arbitrary and capricious rules to be trusted, a CA should not be "trusted" by the browser, but only able to sign certificates for domains registered to it.
- greyface- 1y agoGood time to note that Buypass offers free certificates over ACME. I have a few of my domains configured to use them instead of LetsEncrypt, just for redundancy and to ensure I have a working non-LE cert source in case LE suffers problems like this over a longer time period. Example OpenBSD /etc/acme-client.conf: authority buypass { api url "https://api.buypass.com/acme/directory" account key "/etc/acme/buypass-privkey.pem" contact "mailto:youremail@example.com" } domain example.com { domain key "/etc/ssl/private/example.com.key" domain full chain certificate "/etc/ssl/example.com.pem" sign with buypass }
- CGamesPlay 1y agoThis is neat. Does cert-manager have facilities to automatically use a fallback ACME provider, so I could automate using this? I'd also accept a pool of ACME providers, but a priority ordering seems ideal. I don't see the functionality listed anywhere, maybe there's some security argument that this is a bad idea?
- attentive 1y agocaddy will auto-issue/renew LE or ZeroSSL depending on availability
- ninjin 1y agoCheers! They look like decent chaps and also outside the US for some additional certificate diversity. Are there other trustworthy Acme issuers out there? A pity that acme-client(1) does not allow for fallbacks, but I will add a mental note about it being an easy enough patch to contribute if I ever find the time.
- grodriguez100 1y agoZeroSSL works very well for me. I found it because it is now the default for the acme.sh client.
- sugarpimpdorsey 1y agoI'm sure those six-day lifetime certificates will work out real nice.
- ocdtrekkie 1y agoI think I am going to become a fan of shorter certificate lifetimes because as soon as the chuckleheads in the CAB truly break the Internet on the level they are pushing for, the sooner we get to discard the entire PKI dumpster fire.
- NooneAtAll3 1y agowhat's the alternative to PKI?
- dylan604 1y agono alternative. just eliminate the thing not liked. it's called being DOGEd
- greyface- 1y agohttps://en.wikipedia.org/wiki/Decentralized_identifier https://en.wikipedia.org/wiki/Decentralized_identifier
- jtchang 1y agoSo basically you trust something because you have a long chain of assurances that you trusted it before? Kinda like certificate pinning.
- haiku2077 1y agoCertainly something a hell of a lot simpler then x509 - and without assumptions from the 1990s hardcoded into it
- cpach 1y ago
- phillipseamore 1y agoAlternatives are available: https://zerossl.com/ https://zerossl.com/ (90 days) https://www.buypass.com/ https://www.buypass.com/ (180 days)
- deleted 1y ago[deleted]
- adamsiem 1y agoI thought I got rate-limited. Bad timing to spin up a new service.
- Kholin 1y agoLet's Encrypt stopped its certificate expiration email notification service a while ago, and I hadn't found a replacement yet. As a result, I didn't receive an expiration notice this time and failed to renew my certificate in advance. The certificate expired today, making my website inaccessible. I logged into my VPS to renew it manually, but the process failed every time. I then checked my cloud provider's platform and saw a notification at the top, which made me realize the problem was with the certificate provider. A quick look at Hacker News confirmed it: Let's Encrypt was having an outage. I want to post this news on my website, but I can't, because my site is down due to the expired certificate.
- compumike 1y agoOof, you're right, that's rough that it's so soon after they discontinued their email service! I wrote this blog post a few weeks ago: "Minimal, cron-ready scripts in Bash, Python, Ruby, Node.js (JavaScript), Go, and Powershell to check when your website's SSL certificate expires." https://heiioncall.com/blog/barebone-scripts-to-check-ssl-certificate-expiration https://heiioncall.com/blog/barebone-scripts-to-check-ssl-ce... which may be helpful if you want to roll your own. (Disclosure: at Heii On-Call we also offer free SSL certificate expiration monitoring, among other things.)
- mixdup 1y agoThey have been communicating the ending of the email notices for quite a while and have been telling users that you should have some other monitoring in place to avoid just this situation
- CamperBob2 1y agoAlso, beware of the leopard.
- jojobas 1y agoIf you didn't see their sunset notification emails you wouldn't have seen your cert expiration email either.
- pgporada 1y agoIt's DNS, we're working on it. Sorry, thank you for bearing with us.
- Titan2189 1y agoIt's not DNS There's no way it's DNS It was DNS
- deadbabe 1y agoFive stages of DNS outage: 1. Denial: It’s not DNS. 2. Anger: What the fuck is it! 3. Bargaining: Maybe it’s a firewall, or Cloudflare! 4. Depression: We’ve checked everything… 5. Acceptance: It’s DNS.
- trinsic2 1y agoLOL. I just went though this the other day. my site was intermittently non-accessible. DNS was the last thing I thought it was until I ran a crawler on my site and spotted some 404 errors. Found that my non-www. url was pointed at the wrong IP and I forgot to update it when I transfered my domain to a new host.
- woleium 1y agoThat ttl is a killer, eh?
- senectus1 1y agowhoa whoa whoa.. slow down! you dont just leap to "It's DNS"... you have to try to blame everything else first before you get to DNS. it's like foreplay!
- dylan604 1y agowhen all of the interns have jumped around the corner before the blame hammer was wielded, you have to move to the next item on the list
- 1y ago
- skluug 1y agoLetsNotEncrypt. zing!
- wnevets 1y agoThis is the first time I remember something like this ever happening with LetsEncrypt
- pgporada 1y agoIt's not, we've had multi-hour long outages before. We learn from our mistakes, adapt, and continue on.
- benlivengood 1y agoHopefully the thundering herd when service is restored doesn't knock things offline again. I know LE designs for huge throughput (something like 3X total outstanding certificates in 24 hours, at one point) and the automated client recommendations for backoff are pretty good, but there will be a lot of manual applications/renewals I'm sure.
- deleted 1y ago[deleted]
- bravetraveler 1y agoWell, that does it: certificate lifetimes are even shorter now. If only the same zest applied to probes