7 ms·
Let's Encrypt Acme API Outage
- agwa 3y agoThis is because I discovered that Let's Encrypt was issuing non-compliant certificates: https://bugzilla.mozilla.org/show_bug.cgi?id=1838667 https://bugzilla.mozilla.org/show_bug.cgi?id=1838667
- rob-olmos 3y agoDo you have a blog post or writeup on how you discovered that? Thanks!
- agwa 3y agoThis all happened less than 2 hours ago, but a quick summary is that my Certificate Transparency monitor, Cert Spotter (https://sslmate.com/certspotter https://sslmate.com/certspotter) performs various sanity checks on every certificate that it observes. At 15:41 UTC today, I started getting alerts that certificates from Let's Encrypt were failing one particular check. I quickly emailed Let's Encrypt's problem reporting address, and Let's Encrypt promptly suspended issuance so they could investigate. I've lost count of how many CAs I've detected having this particular problem, so perhaps it is time to blog about it (https://www.agwa.name/blog https://www.agwa.name/blog if you're interested).
- conroydave 3y agothis is why i will always love hacker news. thank you
- deleted 3y ago[deleted]
- mardifoufs 3y agoThat's awesome!! I wonder if let's encrypt runs sanity checks before/after issuing certs too?
- agwa 3y agoThey "lint" certificates before issuance, as do most CAs. However, I don't think any linters check for this problem, as it requires access to more than just the certificate (the linter would need access to either the precertificate or a database of Certificate Transparency log keys).
- pabs3 3y agoAre these linters open source?
- agwa 3y agoYup, the two most popular are: https://github.com/zmap/zlint https://github.com/zmap/zlint https://github.com/certlint/certlint https://github.com/certlint/certlint They each have their strengths and weaknesses, so CAs are advised to use both.
- mcpherrinm 3y agoWe will add a lint to Boulder for precertificate and certificate correspondence to ensure this class of problem never happens again. It would be nice to add this to Zlint, but we'd need a new interface that could be given both a precertificate and certificate to co-lint. Other than this one correspondence check, I'm not sure if there's any other lints that would fit that pattern.
- dopamean 3y agoThis iso so awesome. Thank you for sharing. I hope you do write about that problem. I'd love to learn something new.
- toomuchtodo 3y agoCurious people contributing to the ongoing functioning of critical systems at scale. Thank you for your effort! https://xkcd.com/2347/ https://xkcd.com/2347/
- mholt 3y agoI would love to read a blog of yours with more information.
- danShumway 3y agoI will also throw out a quick vote that I'd be interested in reading a blog post about it.
- VWWHFSfQ 3y agoSounds like somebody didn't properly seed their random number generator
- agwa 3y agoThe problem is actually WAY more subtle, and pretty hard to understand unless you really get in the weeds of Certificate Transparency and certificate policy, but I'll give a shot at providing a concise explanation. Let's Encrypt has produced two signed artifacts with the same serial number: 1. A precertificate: https://api.certspotter.com/v1/certs/22700bd0d70ac5790e6ae5b5f10afc998bb062b4fb1a153fd71b3f1b98fb8b00.pem https://api.certspotter.com/v1/certs/22700bd0d70ac5790e6ae5b... 2. A certificate: https://api.certspotter.com/v1/certs/c0916d24ac8844522b3695093fb66b3e9ff71b5f25b6d563030113ed03833f0b.pem https://api.certspotter.com/v1/certs/c0916d24ac8844522b36950... A precertificate is not a certificate, but it implies the existence of a corresponding certificate which can be constructed by applying an algorithm to the precertificate. Let's Encrypt intended to create a precertificate which would result in (2) when applying the algorithm to (1). Unfortunately, applying the algorithm to (1) results in a different certificate, (3), presumably because of some bug in Let's Encrypt. Since (2) and (3) have the same serial number, it's a violation of the prohibition against duplicate serial numbers. An easier-to-understand description of the problem is that Let's Encrypt was producing precertificates that didn't match the final certificate, but the compliance violation is duplicate serial numbers, which is why I worded my compliance bug the way I did.
- VWWHFSfQ 3y agoFascinating! What kind of effects can this have in real life? Are they going to have to revoke the affected certificates?
- agwa 3y agoThey will indeed need to revoke the affected certificates within 5 days, per the Baseline Requirements. Also, the affected certificates won't be accepted by Certificate Transparency-enforcing browsers (Chrome and Safari) because of the precertificate mismatch.
- iso1631 3y agoSo you can legitimately put "broke the internet" on your resume :D
- dylan604 3y agoThat would be impressive except for all of the AWS us-east-1 engineers that can claim the same thing
- natebc 3y agoThey didn't do it all by themselves though!
- dylan604 3y agoLike you don't have things on your resume that your team did and not just you
- gumby 3y agoOne thing I like about the computing world is that people put "wrote the package that frobbed jpegs that in production frobbs over 1 million jpegs per hour". Anything less specific means "I was somewhere in the building when this was written and deployed". When I worked in pharma people would say something like "I joined a program targetting neurology early in the preclinical phase and developed assays, until first-in-human four years later. Started as senior lab technician and departed as a junior assistant director for preclinical QC." Took me years to understand how the sociology, regulatory dynamics, and science of the two fields legitimately resulted in these utterly different approaches.
- deleted 3y ago[deleted]
- StayTrue 3y agoLooks like they're back online with a fix.
- mholt 3y agoRegular reminder that the best ACME clients will fall back to other CAs if one is down. For example caddy does this. (Disclosure yada yada)
- yjftsjthsd-h 3y ago> the best ACME clients will fall back Are there others that do it, or are you just saying that yours is the best?
- xen2xen1 3y agoI used Caddy and was just wondering if my stuff would be broke, so the GP comment was useful to me, and snarky as your reply was.
- yjftsjthsd-h 3y agoSure, there are a handful of useful points there (off the top of my head: it's possible to work around this failure by using multiple CAs, multiple free ACME CAs exist, caddy implements this solution). I'm just 1. slightly frustrated that caddy's author never seems to miss a chance for self-promotion (at least he's started alluding to the fact that it's his project), and 2. actually curious whether any other ACME clients are implementing that fallback.
- mholt 3y ago> I'm just 1. slightly frustrated that caddy's author never seems to miss a chance for self-promotion (at least he's started alluding to the fact that it's his project), Sorry that I happen to be the author. It's really not about that though -- it just matters that an ACME-native HTTPS server exists. We need more integrated fully-native ACME clients. > actually curious whether any other ACME clients are implementing that fallback. There are at least one or two others. I don't recall which ones at the moment but I think Certify the Web may be one. Edit: mod_md is another apparently!
- yjftsjthsd-h 3y agoOn the bright side, that's actually one of the lower-impact things to have an outage on, IMO; if you're using it the recommended way, an outage would only really affect new certs, with older certs just getting renewed slightly later.
- johncolanduoni 3y agoIf you’re onboarding new users into something like a SaaS or PaaS it could be a bigger deal.
- 8organicbits 3y agoWhat's the impact of an outage like this? ACME renewals should happen daily starting 30 days before expiry so no one should have had a cert expire due to this. New certificates wouldn't have been issued, so that's impact, although I suspect most new certs aren't taking traffic immediately (i.e. setting up a new server).