21 ms·
Issue with TLS-ALPN-01 Validation Method
- hafkensite 5y agoMy traefik setup is affected, should not be to difficult to refresh. It's automated anyway
- mastax 5y agoWonder how many ACME deployments check for revocation, rather than just being on an infrequent cron job? What proportion of affected certificates will be automatically renewed with no effort? Looking at a few docs, probably not many. In any case there isn't (?) an in-band way to tell the clients that the cert is going to be revoked before it is revoked, so there would be some disruption.
- Kesseki 5y agoThere's a plan to make this information available to clients in the future: https://datatracker.ietf.org/doc/draft-aaron-acme-ari/ https://datatracker.ietf.org/doc/draft-aaron-acme-ari/
- Ayesh 5y agoYeah, further, I doubt many certificates were issued from an account key belonging to a email address that people monitor often, if not at all.
- petecooper 5y agoI hadn't considered this. Am I in the minority by having a legit, monitored email account for my ACME certs?
- kiwijamo 5y agoI thought having a proper email account was what most people do. Companies probably use a role-based address e.g. certmaster@myco.com so the email goes to whoever is responsible for it today.
- octoberfranklin 5y agoWebPKI certificate revocation doesn't work anyways. It fails in exactly the case where TLS is needed: MITM. All certificate revocation-checking schemes "fail open" and proceed happily on their way if the MITM blocks their communications with the revocation lists. If you somehow don't have to worry about MITM you don't need anything remotely close to the complexity of TLS. Certificate revocation is mostly security theater.
- vbezhenar 5y agoThat's issue with buggy clients which should not proceed if CRL is not available. Not issue with PKI per se.
- octoberfranklin 5y agoIf they didn't fail open, every time a CA's website went down every single website that used their certificates would go offline as well. You can imagine the DDOS-ransomers licking their lips at this possibility. No, "fail open" has always been the only possible way to implement this. Which is why it's a broken idea from the start.
- vbezhenar 5y ago> If they didn't fail open, every time a CA's website went down every single website that used their certificates would go offline as well. That's not correct. OCSP stamps exist to prevent that kind of a problem.
- zauguin 5y agoOCSP always seemed a bit absurd to me: Instead of sending a OCSP stamp, the CA could also issue a very short lived certificate on demand. It would have the same effect of asserting that the CA currently considers the server to be verified and it doesn't need a separate format.
- cmeacham98 5y ago
- mholt 5y agoCaddy does. https://community.letsencrypt.org/t/questions-about-renewing-before-tls-alpn-01-revocations/170449/21?u=mholt https://community.letsencrypt.org/t/questions-about-renewing... And this is one reason why I keep advocating for certificate automation to be built into services/apps, rather than patched on the outside with duck tape. I look forward to the day when cert lifetimes are regularly about as short as OCSP responses. Then we can possibly do away with OCSP entirely.* (* I am of the opinion that revocation is fundamentally broken for Web PKI and it should be phased out in favor of short cert lifetimes. You may disagree and that's fine, but I'm happy to discuss why if you're interested.)
- stefan_ 5y agoThe way LE and others keep breaking this process and the tools around it is certainly not a great endorsement for having it integrated into a service.
- jabiko 5y agoCan you give some example of the kind of breakage your experienced?
- kijin 5y agoNot OP, but here are some things I've personally experienced: 1. Supposedly more secure challenge types such as TLS-ALPN-01 are far from stable, as the current incident shows. Your cert can be revoked at any time through no fault of your own. After being burned by TLS-SNI-01 the last time, now I refuse to use anything other than plain old HTTP-01 and DNS-01. 2. As soon as the version of the Linux distro I was using (not in my power to change!) reached EOL, certbot suddenly refused to renew, despite the fact that I'd been using more or less the same version of Python and certbot for a number of years and the HTTP-01 challenge requires nothing fancy at all. Why does everyone these days insist on making ops decisions for other people? 3. On a server with existing nginx virtual hosts, certbot injects configuration directives including stuff the nginx team officially recommends against, such as `if` statements. It frequently breaks existing configuration such as rewrites and redirects. After seeing this a number of times, the only conclusion I can make is that certbot has no idea how to manipulate nginx config files. 4. If I have multiple domains pointing at the same application, and remove one of them at a later time, certbot is oblivious and repeatedly fails trying to renew the certificate that now contains an invalid domain. Again, certbot doesn't know how to work with nginx. Maybe 3 and 4 can be improved if ACME was integrated as a proper nginx module instead of certbot trying to change things from the outside. My experience as a whole, however, makes me feel that the LE/certbot teams are rather cavalier about the commitment to stability they need to make if they really want to become an essential part of the world's internet infrastructure. If you want to be paternalistic about managing TLS for people who don't know how to do it, at least try to do it properly!
- LukeLambert 5y agoThis is the second security issue with a TLS-based challenge [1]. This was a good reminder to switch to the HTTP challenge for the one remaining server I had that was affected. [1] https://letsencrypt.org/docs/challenge-types/#tls-sni-01 https://letsencrypt.org/docs/challenge-types/#tls-sni-01
- octoberfranklin 5y agoThat might work for you, but ALPN needs to exist because there's more to the Internet than just HTTP, and TLS can be used for those non-HTTP protocols. Some of those protocols are more fundamental than HTTP, and making them depend on HTTP would create a circular dependency. HN is choking again, so I must reply with edits *sigh* @tialaramex, you're confusing policies of one CA (LE) with the ALPN protocol. Lets Encrypt isn't the only CA out there. Even so, you can do TLS-ALPN on any port. You can do TLS-ALPN on port 443 without using the HTTP protocol in any way. To ALPN, 443 is just an arbitrary number, like the IP address of Lets Encrypt's server. > If you actually want certificate issuance unrelated to web servers you should either hook up a web server Good heavens, no.
- tialaramex 5y agoAlthough "TLS can be used for those non-HTTP protocols" the tls-alpn-01 validation can only be used on the authorised ports, which for Let's Encrypt is port 443, aka HTTPS. Now, Let's Encrypt would technically be allowed to enable this validation on a few other ports, 80 (HTTP), 25 (SMTP) and 22 (SSH) under current Baseline Requirements, but understandably they have no plan to do that. If you actually want certificate issuance unrelated to web servers you should either hook up a web server explicitly for issuance or use DNS proof of control.
- rad_gruchalski 5y ago> Good heavens, no. You skipped the second part of the argument. And what "no" means in that context. What's your alternative?
- xg15 5y ago> Good heavens, no. It's not exactly the most elegant solution, but I don't understand the aversion either. A "web server" that is only intended to serve the challenge file can be as simple as a thread that writes a static blob of bytes to a socket. That's nc -l stuff. If you're already modifying your TLS backend to understand the ALPN challenge, I don't see why it would be that hard to add logic for one specific GET/200 OK pair.
- therealunreal 5y agoNo action necessary for Caddy: https://twitter.com/caddyserver/status/1486226944597233664 https://twitter.com/caddyserver/status/1486226944597233664
- flipbrad 5y agoCaddy rightly gets a lot of love from its users - I've been a super satisfied user for a while. Rock solid reverse proxy to some selfhosted services.
- juriansluiman 5y agoAs my Traefik setup is affected, I cleared the `acme.json` and let Traefik get new certificates for all services. Seems LE is pretty busy right now, got time outs flying around every where.
- deleted 5y ago[deleted]
- pstrike 5y ago
- Aissen 5y agoSmall feedback for the letsencrypt folk: I got the email saying that I have two ACME accounts ids affected. It would have been nice to know which domains are (even if it's just the first ~10 or so per account).
- hannob 5y agoThey made the same mistake with the SNI deprecation and got plenty of feedback about it... It's disappointing that they haven't learned from this.
- sebiw 5y agoI understand your pain. Maybe keep in mind what operation they are running (complexity and scale) and that feedback may take time to implement. To put this into perspective think about what replacing certificates (or say obtaining new certificates) felt like before Let's Encrypt an ACME were a thing. ;-)
- Kesseki 5y agoThe full list of affected certificates and domains is now available: https://community.letsencrypt.org/t/170449/ https://community.letsencrypt.org/t/170449/
- hannob 5y agoCan anyone make sense of what they're trying to tell there? They found some issue ("irregularities") and made 2 changes, but the changes are merely restricting the TLS version to 1.2 and deprecating an old OID identifier. While TLS < 1.2 certainly is not ideal, I don't see how this would impact the ACME validation, and the old OID should be irrelevant as well. (I have been somewhat concerned about the security properties of the acme/alpn validation for unrelated other reasons, but haven't been able to pin that down to a specific threat - notably the RFC implies that the security is improved due to strict ALPN validation, which in practice usually does not happen.) Update: RFC 8737 (the ALPN validation method) says "ACME servers that implement "acme-tls/1" MUST only negotiate TLS 1.2". So maybe this is "just" a policy issue?
- walrus01 5y agoThe population of browsers and things that don't understand TLS1.2 is miniscule now, so there should be no impact in disabling tls1.1 and below anywhere possible.
- YPPH 5y agoBrowsers aren't to the point here, it is referring to servers seeking to verify control of a domain, by conducting a “acme-tls/1” handshake initiated by the Let's Encrypt issuance server. If such a server only supports TLS 1.1, then TLS-APLN-01 validation will fail after this change is implemented.
- walrus01 5y agoNobody should be running tls1.1 only (and incapable of opportunistically negotiating for 1.2 or 1.3 instead) on their public facing httpd in 2022, I disabled everything below 1.2 on some rather high traffic websites several years ago with zero impact.
- perlgeek 5y agoIt seems the reasoning here is: the TLS handshake might have used an insecure TLS version, and so they cannot be sure that the handshake worked the way they thought, and so the certificates could have been issued to the wrong party. I don't have a deep understanding of the TLS-ALPN-01 validation nor of the vulnerabilities they might be concerned about, but that would be the only reason for revoking certificates (unless it's more of a political statement, "we revoke certs when we screwed up!").
- xg15 5y agoWhat I don't quite get with all the certificate automation: Doesn't this all effectively just shift the "source of truth" to DNS? Back when certificates were issued manually, a CA was also verifying that the requesting party was actually who they were claimed to be IRL - hence EV certificates and all that. What LE and friends verify on the other hand is simply that the entity that requests a certificate also controls the DNS entry at that point in time - or at least controls some of the servers that are listed in the A/AAAA records. For one of the infamous Authoritarian Governments, it should be no problem at all to obtain an LE certificate for any domain under their ccTLD. Just use the DNS challenge, then instruct the country's registrar to change the DNS record for the domain of interest. Isn't that a massive centralisation compared to the old system?
- discreditable 5y agoIt was always DNS. Unless you are getting EV the CAs usually verify ownership via email. Email can go anywhere the current MX record in DNS says it goes.
- LeonM 5y agoThis is why for EVs most CAs also do phone validation. For stuff like Verified Mark Certificates (which is used for BIMI), it goes much further than that. VMCs are like EVs on steroids. HN crowd can sometimes react very hostile towards having to pay anything at all for certificates, but there are real costs in such validations.
- r1ch 5y agoThat "manual verification" back in the day was an email verification link, so again, back to relying on DNS. I don't think there's a way around it, but at least with certificate transparency we'll know if it happens.
- Genbox 5y agoYou are right. The EV validation process (I am EV validated at 3 major authorities) does involve sending paystubs, identification with photos and an interview. Let's Encrypt is not a solution for trust. It is an attempt at getting as many people to adopt HTTPS as possible. There used to be 3-4 free certificate authorities which all had their problems (processes, security, uptime etc.) and Lets Encrypt is outperforming them all. We still need to understand issues around identity, which can only be solved with verification and trust. X509 is encryption keys + trust and LE has much weaker trust guarantee than an EV.