11 ms·

I exploited TLS-SNI-01 issuing Let's Encrypt SSL-certs for any domain (2018)

240 points by yread 8y ago

CaliforniaKarl 8y ago
Let's Encrypt did end up disabling TLS-SNI-01 on January 12, 2018: https://community.letsencrypt.org/t/tls-sni-challenges-disabled-for-most-new-issuance/50316 https://community.letsencrypt.org/t/tls-sni-challenges-disab... And as per https://community.letsencrypt.org/t/important-what-you-need-to-know-about-tls-sni-validation-issues/50811 https://community.letsencrypt.org/t/important-what-you-need-..., it looks like TLS-SNI-01 and TLS-SNI-02 will be dropped, and moved to TLS-SNI-03 that addresses the issue. So, as long as your ACME client is up-to-date, you should be fine! And if your ACME client isn't up-to-date, then you're either using a different method or you're probably not getting certs.
- thro_away_n 8y ago
  You get an email like this: Hello, Action may be required to prevent your Let's Encrypt certificate renewals from breaking. If you already received a similar e-mail, this one contains updated information. Your Let's Encrypt client used ACME TLS-SNI-01 domain validation to issue a certificate in the past 60 days. Below is a list of names and IP addresses validated (max of one per account): example.com (x.x.x.x) on 2018-12-05 TLS-SNI-01 validation is reaching end-of-life. It will stop working temporarily on February 13th, 2019, and permanently on March 13th, 2019. Any certificates issued before then will continue to work for 90 days after their issuance date. You need to update your ACME client to use an alternative validation method (HTTP-01, DNS-01 or TLS-ALPN-01) before this date or your certificate renewals will break and existing certificates will start to expire. Our staging environment already has TLS-SNI-01 disabled, so if you'd like to test whether your system will work after February 13, you can run against staging: https://letsencrypt.org/docs/staging-environment/ https://letsencrypt.org/docs/staging-environment/ If you're a Certbot user, you can find more information here: https://community.letsencrypt.org/t/how-to-stop-using-tls-sni-01-with-certbot/83210 https://community.letsencrypt.org/t/how-to-stop-using-tls-sn... Our forum has many threads on this topic. Please search to see if your question has been answered, then open a new thread if it has not: https://community.letsencrypt.org/ https://community.letsencrypt.org/ For more information about the TLS-SNI-01 end-of-life please see our API announcement: https://community.letsencrypt.org/t/february-13-2019-end-of-life-for-all-tls-sni-01-validation-support/74209 https://community.letsencrypt.org/t/february-13-2019-end-of-... Thank you, Let's Encrypt Staff
  - opless 8y ago
    Hopefully Ubuntu has updated certbot to support this.
    - GuyPostington 8y ago
      If not, installing from pip is always an option. `pip install --upgrade --user certbot`
      - schoen 8y ago
        We would really suggest using certbot-auto instead because it will create a venv for you so that you don't get version conflicts elsewhere. (It does use pip behind the scenes, but in a venv.)
        GuyPostington 8y ago
        That's a good point, thank you.
    - gamegod 8y ago
      They have updated, just `apt-get upgrade`.
    - rlpb 8y ago
      Ubuntu Server developer involved with Certbot here. I have posted elsewhere in this discussion with details.
- mholt 8y ago
  "TLS-SNI-03" ended up being TLS-ALPN-01 and has been in production for some time: https://community.letsencrypt.org/t/tls-alpn-validation-method/63814 https://community.letsencrypt.org/t/tls-alpn-validation-meth...
Ajedi32 8y ago
FWIW, this has long since been patched and TLS-SNI is no longer considered a valid method for ACME. It's being replaced with TLS-ALPN which doesn't have this problem: https://community.letsencrypt.org/t/tls-alpn-validation-method/63814 https://community.letsencrypt.org/t/tls-alpn-validation-meth...
- djrogers 8y ago
  Long since? TLS-SNI-01 and 02 were only disabled only a few weeks ago, and they haven't been officially removed from the ACME spec, and the replacement - TLS-SNI-03 - hasn't been finalized yet. I think you're conflating LetsEncrypt with ACME here. LE turned off the 2 methods, and added a different tls validation method last year, but neither of those are directly related to fixing the ACME spec.
  - mholt 8y ago
    TLS-ALPN-01 is the successor to TLS-SNI, and it has been in production at Let's Encrypt for some time now. https://community.letsencrypt.org/t/tls-alpn-validation-method/63814 https://community.letsencrypt.org/t/tls-alpn-validation-meth... TLS-SNI has been disabled for almost a year for new issuances and the final nail in the coffin is to turn it off for renewals which is happening soon. https://community.letsencrypt.org/t/important-what-you-need-to-know-about-tls-sni-validation-issues/50811 https://community.letsencrypt.org/t/important-what-you-need-...
  - zjs 8y ago
    > TLS-SNI-01 and 02 were only disabled only a few weeks ago, and they haven't been officially removed from the ACME spec, ... Based on the repository's history, TLS-SNI-01 and TLS-SNI-02 appear to have been removed from the spec approximately a year ago: https://github.com/ietf-wg-acme/acme/commit/cfe118734bab3f6b7417463aad7e7faa8709df12 https://github.com/ietf-wg-acme/acme/commit/cfe118734bab3f6b...
peterwwillis 8y ago
For a while I found it funny that the web's security is based on the strength of (1) new standards from Let's Encrypt, and (2) the quality of their codebase. It's been shown multiple times that they have failed in both of these, yet nobody has taken them to task as failing to uphold the responsibility that CAs have to keep the web secure. (https://community.letsencrypt.org/c/incidents https://community.letsencrypt.org/c/incidents) At first I thought that their biggest issue was properly verifying that a domain owner is requesting a certificate (they don't do that, but anyway..). They actually have multiple categories of security holes relating to web pki, because it's a pretty big sandbox. The holes affecting LE include everything from OCSP support, to actually not issuing any certificates during Google outages (yes, if Google is down, LE is down, as well as other CAs, and it's happened multiple times), to exposing the personal information of LE users, to bugs in their protocol allowing any cert to be issued, to mis-handling blocklists, leaking API keys, and so on. Basically, a litany of different security holes could at any time compromise the security of the whole damn web. The thing is, this isn't specific just to LE. All these things and more probably affect every CA. LE is just more experimental and new, using less robust infrastructure and technology, and their "growing pains" provide a much more visible and much wider attack surface for abusing PKI. I don't think the industry organizations connected to CAs care enough to put in stringent security practices in place. But I do think it's about time to start producing these standards. It's probably going to be up to hobbyists and privacy aficionados to do this work, because they're the ones who care enough. I'm also surprised that governments aren't more active in enforcing more stringent security practices, seeing as they're affected just as much. Interestingly, following the CT Policy mailing list is a good way to be told about holes in the current CA system: https://groups.google.com/a/chromium.org/d/msg/ct-policy/_csiMYrwsxc/2S6bMnSCDwAJ https://groups.google.com/a/chromium.org/d/msg/ct-policy/_cs...
- annonch 8y ago
  Hmm you may be right in general but this post is about a vulnerability in a specification adopted by LE not a standard designed by LE.
  - schoen 8y ago
schoen 8y ago
I work on Let's Encrypt and Certbot and I'll offer the following summary for people who aren't familiar with the history of this. TLS-SNI-01 is one of several validation methods for getting a Let's Encrypt certificate. It was originally the only method that worked on port 443 (other methods use port 80 or ask you to create DNS records). In January 2018, Frans Rosén (the author of the article linked here) discovered that, on some shared hosting environments, one shared hosting customer could pass TLS-SNI-01 challenges for a different customer's domain. This problem only affects users of particular shared hosting services, but there was no apparent way that Let's Encrypt could get every shared hosting provider to fix this. (Let's Encrypt already knew about a related problem in a proposed-but-never-implemented validation method called HTTPS-01, but had missed the issue that Rosén discovered.) Let's Encrypt then temporarily disabled TLS-SNI-01 entirely, and later allowed it for certificate renewals but not for initial issuance, as well as for specific hosting providers that had specifically confirmed that they were not affected by the vulnerability. At the same time, the eventual deprecation of TLS-SNI-01 was announced. In preparation for the deprecation, the Certbot client improved its support for HTTP-01 (the challenge that uses port 80), and, in recent releases, switched to preferring HTTP-01 over TLS-SNI-01 even if the certificate authority offers both. (Certbot originally preferred TLS-SNI-01 over HTTP-01.) This change means that recent renewals for people who have moderately up-to-date versions of Certbot will default to HTTP-01 and simulate what will happen after TLS-SNI-01 isn't available at all. Recently, Let's Encrypt has also been e-mailing people who performed recent renewals using TLS-SNI-01 and warning them that this option is going to be eliminated permanently in March. There is a newer validation method that works on port 443 called TLS-ALPN-01; this is supported by Let's Encrypt but not in Certbot. Some other clients do support it, although they may require that you shut down your web server temporarily. https://community.letsencrypt.org/t/which-client-support-tls-alpn-challenge/75859/2 https://community.letsencrypt.org/t/which-client-support-tls... Some people have been frustrated by this change because they have blocked port 80 entirely. Let's Encrypt has published a document noting that this isn't a good idea; in particular, it doesn't protect against active SSL stripping attacks. https://letsencrypt.org/docs/allow-port-80/ https://letsencrypt.org/docs/allow-port-80/
- wnevets 8y ago
  >Recently, Let's Encrypt has also been e-mailing people who performed recent renewals using TLS-SNI-01 and warning them that this option is going to be eliminated permanently in March. So this is what that email was about.
numbsafari 8y ago
While it's not a mitigation, something you should consider doing is using a certificate transparency[0] monitor[1] for your domains. I use the crt.sh service to monitor my domains. It provides RSS feeds (e.g. [3]) that you can hook up to your own monitoring infrastructure. I have this piped into a slack channel, so that any time a certificate is issued, I get a warning. Since I am using LE for certs, I will often cross-reference that issuance with the logs from our LE refresh bot. A future step would be to do that automatically, but this is sufficient for now. [1] https://www.certificate-transparency.org/ https://www.certificate-transparency.org/ [2] https://crt.sh https://crt.sh [3] https://crt.sh/?q=%25.ycombinator.com https://crt.sh/?q=%25.ycombinator.com
- joombaga 8y ago
  How do they know when a cert is issued for your domain? Edit: I never knew about this! https://tools.ietf.org/html/rfc6962 https://tools.ietf.org/html/rfc6962
  - schoen 8y ago
    The certificate authorities proactively tell public logs, which these services monitor. https://www.certificate-transparency.org/ https://www.certificate-transparency.org/ https://crt.sh/monitored-logs https://crt.sh/monitored-logs
- AdamGibbins 8y ago
  You should also publish CAA records, to whitelist exactly what providers are allowed to issue on behalf of your domain.
whizzkid 8y ago
I would 100% use this article as my CV if I were him. Good work my friend :)
mitchtbaum 8y ago
Where's the CVE? ie. https://www.cvedetails.com/ https://www.cvedetails.com/ ...
- Foxboron 8y ago
  CVEs are traditionally limited to software packages. You wouldn't issue a CVE for the specification nor the cloud vendors.
  - deleted 8y ago
    [deleted]
  - mitchtbaum 8y ago
    "You can't improve what you don't measure.”
    - mitchtbaum 8y ago
      Rather: If you want to improve something, it helps to measure it.
johnchristopher 8y ago
Is traefik auto issuance and renewal still working ?
ohnoesjmr 8y ago
Not sure I understand how this attack works. You ask for cert for foo.com, LE tells you to provide a self-signed cert subjectAlt=foo.bar.acme.invalid. LE resolves foo.bar to some heroku endpoint as the domain lives on heroku, LE connects via TLS without verification and sends foo.bar.acme.invalid as SNI, which heroku routes to a hijacker who asked heroku to route foo bar.acme.invalid requests? Why would LE send the SNI in the first place? I thought the purpose was to prove you own the domain, not cohabit an environment where the domain is hosted? I guess it only allows hijacking certs of domains running on heroku et al envs? Also, what do host headers have to do with this? Presumably this is just a tls handshake test?
- xianb 8y ago
  > Why would LE send the SNI in the first place? I thought the purpose was to prove you own the domain, not cohabit an environment where the domain is hosted? The assumption was that you controlled the domain if you could return the self-signed cert with subjectAlt=foo.bar.acme.invalid when the SNI request for foo.bar.acme.invalid is made to the server your are requesting a cert for. Unfortunately the assumption didn't hold up because hosting providers shared the same routing server across domains and subdomains and those routing servers did not have controls around the subjectAlt domains used for TLS-SNI-01. > Also, what do host headers have to do with this? Presumably this is just a tls handshake test? They don't have anything to do with the weakness. It's mentioned to make the distinction that SNI is used for the cert retrieval to establish the connection and Host-header is used separately to route to the proper backend
  - morpheuskafka 8y ago
    It seems that this arose from a fundamental issue with a lot of internet specifications. The person developing hosts' SNI implementation did not consider SNI to be a form of unique identification, but rather a way to establish that the server was authorized to serve it--as long as the server makes sure to serve the right files, which is a security issue for them to deal with, the certificate can be returned to anyone who requests it, even just to get a 404 page. It's like the janitor randomly trying keys from a ring, doesn't matter whose key is whose as long as it gets him in and he does what he is supposed to. The TLS-SNI-01 developers assumed, presumably based on the implementations they knew/wrote, that SNI was an identifier. When used as originally intended, of course, SNI provides the name of a real certificate desired, not a validation string, and the returned certificate authenticates a pairing of server/IP->domain not validationstring->returned certificate. I'm not sure what the TLS standard documenting SNI actually says is the right interpretation, but unless it clearly says that this might be done in the future, it seems to me like a "hackish" solution that could reasonably be expected to cause a lot of issues. This seems to be a failure of setting clear standards for critical security protocols more than anything else. By contrast, HTTPS-01 and DNS-01 operate using a known authoritative measure, ability to control the actual content of the website. If a bad actor has access to this, its game over. Only EV certs are intended to protect against this. Likewise, TLS-APLN-01 created a new protocol that could not possibly by implemented by accident or by anyone not intending to authorize issuance. It seems like nearly every major security issue (short of hard coding/crypto flaws) involves an assumption or edge case relied upon without consulting relevant standards.
hannob 8y ago
The upcoming deprecation of TLS-SNI-01 (Feb 13th) will catch a few people by surprise, because some Linux distros have been slow in reacting to this. Debian has only recently added an updated certbot to the stretch-updates repo and the normal stable repo still has an old version that defaults to TLS-SNI-01 for apache+nginx. Ubuntu 16.04, which is an LTS version, has an even older version of the letsencrypt software (before it was called certbot) and Ubunbtu doesn't seem to care.
- rlpb 8y ago
  > Ubuntu 16.04, which is an LTS version, has an even older version of the letsencrypt software (before it was called certbot) and Ubunbtu doesn't seem to care. Ubuntu Server developer involved with Certbot here. We do care. Updating letsencrypt/certbot in 16.04 is monumentally difficult because, from a Certbot perspective, February 2016 is prehistoric, and users of an Ubuntu stable release expect not to be regressed for all the different use cases they may have, not all of which we even know about. Updating a stable release is difficult for these reasons normally. Now add in the complication of five different interacting source packages, an upstream project rename, and the need to not regress the behaviour of a key library which must be updated but users may be depending on directly, and hopefully you can see the difficulty of this task. We are working on it though, and you can follow progress here: https://launchpad.net/bugs/1640978 https://launchpad.net/bugs/1640978. I'm still hopeful that this will land in time. In the meantime, if you can't wait, are using 16.04 and must have it now, then you have a number of options: 1) Use certbot-auto as recommended by upstream. 2) Use 18.04, which isn't affected as it shipped with a new-enough certbot package. 3) Try the (experimental) snap: https://forum.snapcraft.io/t/call-for-testing-certbot-lets-encrypt/7990 https://forum.snapcraft.io/t/call-for-testing-certbot-lets-e... (though I don't recommend this for production). Relative to the deb packaging, the snap has been trivial to develop and maintain and the edge channel keeps up with upstream master automatically as long as CI passes (and keeping it passing has been a relative breeze).
walrus01 8y ago
It is worth noting that the version of 'certbot' currently packaged for debian-stable (stretch) is very old, and attempts to use tls-sni-01 as the default method for verification. Strongly recommended to enable stretch-backports and update certbot to v0.28.
joering2 8y ago
So thru my network I see 278 servers I need to patch. This will take forever, or couple days of work. Next time I will simply spend few bucks on an annual certificate instead.
- Dylan16807 8y ago
  A few bucks, or a few bucks multiplied by 278? If you'd be replacing everything with a wildcard because it's easier, you can get a free wildcard from Let's Encrypt too.
- GuyPostington 8y ago
  Or you know, update your packages.
B-Con 8y ago
Rewording (and hopefully simplifying) for my own benefit because I think the OP is a little loose with the details. There are two parts: Part 1: Subdomain takeover 1) DNS for domain X points to service Y (Y=Heroku in this case). 2) Service Y allows anyone to claim an unused domain. 3) Domain X is then claimed by an attacker. 4) Domain X is then served by an attacker. 5) Bonus points if the owner of X has a wildcarded HTTPS cert, so the hijacked subdomain gets HTTPS automatically. So now attacker serves content on a domain they don't own (possibly over HTTPS). This is already a known cloud hosting mis-configuration problem. In this case, investor.example.com for HTTPS was not claimed on Heroku but DNS was pointed to Heroku, so it was vulnerable to subdomain takeover. However, the author only had HTTPS access and example.com did NOT have wildcarded cert, so the author was in an edge case of having control of a subdomain but only over HTTPS with no cert. This prompted them to look for... Part 2: Abusing TLS-SNI-01 validation TLS-SNI-01 works by having LetsEncrypt request a cert with a secret from the specified domain. 1) LE does a DNS lookup of the desired domain. 2) LE connects to that domain using TLS-SNI. SNI specifies a hostname in the pre-encryption request. 3) ACME TLS-SNI-01 has the SNI requested domain in the form foo.bar.acme.invalid. 4) Author claimed the necessary foo.bar.acme.invalid hostname on Heroku (and later AWS). 5) Author initiated a LetsEncrypt cert request for a different domain on the same hosting provider. 6) When LE made the SNI request to that domain it did a DNS lookup and got pointed to the shared hosting provider, but the SNI hostname was of the form foo.bar.acme.invalid which was controlled by the attacker, so it was passed to the attacker's app. 6.a (Side note: Because there was no valid cert offered by the hosting provider, I'm guessing the front-ends passed the SNI request through to the backend app. Because if there was a cert configured for the Heroku app then I think the front-ends just serve the cert directly without checking with the backend app.) 7) The attacker then finished the LE cert negotiation and got a cert on behalf of the other domain. I'm a little fuzzy on exactly how choosing the domain in (3) and (6) works since the spec[0] doesn't seem to match what the blog post shows, but I'm sure it works somehow. Corrections/clarifications welcome. Related specs: [0] https://tools.ietf.org/html/draft-ietf-acme-acme-01#page-40 https://tools.ietf.org/html/draft-ietf-acme-acme-01#page-40 [1] https://tools.ietf.org/html/rfc6066#page-6 https://tools.ietf.org/html/rfc6066#page-6
kpcyrd 8y ago
The title on hackernews is highly misleading, there's quite a difference between "for any domain" and "for any domain using shared hosting".

240 points by yread 8y ago

Let's Encrypt did end up disabling TLS-SNI-01 on January 12, 2018: https://community.letsencrypt.org/t/tls-sni-challenges-disabled-for-most-new-issuance/50316 https://community.letsencrypt.org/t/tls-sni-challenges-disab... And as per https://community.letsencrypt.org/t/important-what-you-need-to-know-about-tls-sni-validation-issues/50811 https://community.letsencrypt.org/t/important-what-you-need-..., it looks like TLS-SNI-01 and TLS-SNI-02 will be dropped, and moved to TLS-SNI-03 that addresses the issue. So, as long as your ACME client is up-to-date, you should be fine! And if your ACME client isn't up-to-date, then you're either using a different method or you're probably not getting certs.

thro_away_n 8y ago
You get an email like this: Hello, Action may be required to prevent your Let's Encrypt certificate renewals from breaking. If you already received a similar e-mail, this one contains updated information. Your Let's Encrypt client used ACME TLS-SNI-01 domain validation to issue a certificate in the past 60 days. Below is a list of names and IP addresses validated (max of one per account): example.com (x.x.x.x) on 2018-12-05 TLS-SNI-01 validation is reaching end-of-life. It will stop working temporarily on February 13th, 2019, and permanently on March 13th, 2019. Any certificates issued before then will continue to work for 90 days after their issuance date. You need to update your ACME client to use an alternative validation method (HTTP-01, DNS-01 or TLS-ALPN-01) before this date or your certificate renewals will break and existing certificates will start to expire. Our staging environment already has TLS-SNI-01 disabled, so if you'd like to test whether your system will work after February 13, you can run against staging: https://letsencrypt.org/docs/staging-environment/ https://letsencrypt.org/docs/staging-environment/ If you're a Certbot user, you can find more information here: https://community.letsencrypt.org/t/how-to-stop-using-tls-sni-01-with-certbot/83210 https://community.letsencrypt.org/t/how-to-stop-using-tls-sn... Our forum has many threads on this topic. Please search to see if your question has been answered, then open a new thread if it has not: https://community.letsencrypt.org/ https://community.letsencrypt.org/ For more information about the TLS-SNI-01 end-of-life please see our API announcement: https://community.letsencrypt.org/t/february-13-2019-end-of-life-for-all-tls-sni-01-validation-support/74209 https://community.letsencrypt.org/t/february-13-2019-end-of-... Thank you, Let's Encrypt Staff
- opless 8y ago
  Hopefully Ubuntu has updated certbot to support this.
  - GuyPostington 8y ago
    If not, installing from pip is always an option. `pip install --upgrade --user certbot`
    - schoen 8y ago
      We would really suggest using certbot-auto instead because it will create a venv for you so that you don't get version conflicts elsewhere. (It does use pip behind the scenes, but in a venv.)
      - GuyPostington 8y ago
        That's a good point, thank you.
  - gamegod 8y ago
    They have updated, just `apt-get upgrade`.
  - rlpb 8y ago
    Ubuntu Server developer involved with Certbot here. I have posted elsewhere in this discussion with details.
mholt 8y ago
"TLS-SNI-03" ended up being TLS-ALPN-01 and has been in production for some time: https://community.letsencrypt.org/t/tls-alpn-validation-method/63814 https://community.letsencrypt.org/t/tls-alpn-validation-meth...

Ajedi32 8y ago

FWIW, this has long since been patched and TLS-SNI is no longer considered a valid method for ACME. It's being replaced with TLS-ALPN which doesn't have this problem: https://community.letsencrypt.org/t/tls-alpn-validation-method/63814 https://community.letsencrypt.org/t/tls-alpn-validation-meth...

djrogers 8y ago
Long since? TLS-SNI-01 and 02 were only disabled only a few weeks ago, and they haven't been officially removed from the ACME spec, and the replacement - TLS-SNI-03 - hasn't been finalized yet. I think you're conflating LetsEncrypt with ACME here. LE turned off the 2 methods, and added a different tls validation method last year, but neither of those are directly related to fixing the ACME spec.
- mholt 8y ago
  TLS-ALPN-01 is the successor to TLS-SNI, and it has been in production at Let's Encrypt for some time now. https://community.letsencrypt.org/t/tls-alpn-validation-method/63814 https://community.letsencrypt.org/t/tls-alpn-validation-meth... TLS-SNI has been disabled for almost a year for new issuances and the final nail in the coffin is to turn it off for renewals which is happening soon. https://community.letsencrypt.org/t/important-what-you-need-to-know-about-tls-sni-validation-issues/50811 https://community.letsencrypt.org/t/important-what-you-need-...
- zjs 8y ago
  > TLS-SNI-01 and 02 were only disabled only a few weeks ago, and they haven't been officially removed from the ACME spec, ... Based on the repository's history, TLS-SNI-01 and TLS-SNI-02 appear to have been removed from the spec approximately a year ago: https://github.com/ietf-wg-acme/acme/commit/cfe118734bab3f6b7417463aad7e7faa8709df12 https://github.com/ietf-wg-acme/acme/commit/cfe118734bab3f6b...

peterwwillis 8y ago

For a while I found it funny that the web's security is based on the strength of (1) new standards from Let's Encrypt, and (2) the quality of their codebase. It's been shown multiple times that they have failed in both of these, yet nobody has taken them to task as failing to uphold the responsibility that CAs have to keep the web secure. (https://community.letsencrypt.org/c/incidents https://community.letsencrypt.org/c/incidents) At first I thought that their biggest issue was properly verifying that a domain owner is requesting a certificate (they don't do that, but anyway..). They actually have multiple categories of security holes relating to web pki, because it's a pretty big sandbox. The holes affecting LE include everything from OCSP support, to actually not issuing any certificates during Google outages (yes, if Google is down, LE is down, as well as other CAs, and it's happened multiple times), to exposing the personal information of LE users, to bugs in their protocol allowing any cert to be issued, to mis-handling blocklists, leaking API keys, and so on. Basically, a litany of different security holes could at any time compromise the security of the whole damn web. The thing is, this isn't specific just to LE. All these things and more probably affect every CA. LE is just more experimental and new, using less robust infrastructure and technology, and their "growing pains" provide a much more visible and much wider attack surface for abusing PKI. I don't think the industry organizations connected to CAs care enough to put in stringent security practices in place. But I do think it's about time to start producing these standards. It's probably going to be up to hobbyists and privacy aficionados to do this work, because they're the ones who care enough. I'm also surprised that governments aren't more active in enforcing more stringent security practices, seeing as they're affected just as much. Interestingly, following the CT Policy mailing list is a good way to be told about holes in the current CA system: https://groups.google.com/a/chromium.org/d/msg/ct-policy/_csiMYrwsxc/2S6bMnSCDwAJ https://groups.google.com/a/chromium.org/d/msg/ct-policy/_cs...

annonch 8y ago
Hmm you may be right in general but this post is about a vulnerability in a specification adopted by LE not a standard designed by LE.
- schoen 8y ago

schoen 8y ago

I work on Let's Encrypt and Certbot and I'll offer the following summary for people who aren't familiar with the history of this. TLS-SNI-01 is one of several validation methods for getting a Let's Encrypt certificate. It was originally the only method that worked on port 443 (other methods use port 80 or ask you to create DNS records). In January 2018, Frans Rosén (the author of the article linked here) discovered that, on some shared hosting environments, one shared hosting customer could pass TLS-SNI-01 challenges for a different customer's domain. This problem only affects users of particular shared hosting services, but there was no apparent way that Let's Encrypt could get every shared hosting provider to fix this. (Let's Encrypt already knew about a related problem in a proposed-but-never-implemented validation method called HTTPS-01, but had missed the issue that Rosén discovered.) Let's Encrypt then temporarily disabled TLS-SNI-01 entirely, and later allowed it for certificate renewals but not for initial issuance, as well as for specific hosting providers that had specifically confirmed that they were not affected by the vulnerability. At the same time, the eventual deprecation of TLS-SNI-01 was announced. In preparation for the deprecation, the Certbot client improved its support for HTTP-01 (the challenge that uses port 80), and, in recent releases, switched to preferring HTTP-01 over TLS-SNI-01 even if the certificate authority offers both. (Certbot originally preferred TLS-SNI-01 over HTTP-01.) This change means that recent renewals for people who have moderately up-to-date versions of Certbot will default to HTTP-01 and simulate what will happen after TLS-SNI-01 isn't available at all. Recently, Let's Encrypt has also been e-mailing people who performed recent renewals using TLS-SNI-01 and warning them that this option is going to be eliminated permanently in March. There is a newer validation method that works on port 443 called TLS-ALPN-01; this is supported by Let's Encrypt but not in Certbot. Some other clients do support it, although they may require that you shut down your web server temporarily. https://community.letsencrypt.org/t/which-client-support-tls-alpn-challenge/75859/2 https://community.letsencrypt.org/t/which-client-support-tls... Some people have been frustrated by this change because they have blocked port 80 entirely. Let's Encrypt has published a document noting that this isn't a good idea; in particular, it doesn't protect against active SSL stripping attacks. https://letsencrypt.org/docs/allow-port-80/ https://letsencrypt.org/docs/allow-port-80/

wnevets 8y ago
>Recently, Let's Encrypt has also been e-mailing people who performed recent renewals using TLS-SNI-01 and warning them that this option is going to be eliminated permanently in March. So this is what that email was about.

numbsafari 8y ago

While it's not a mitigation, something you should consider doing is using a certificate transparency[0] monitor[1] for your domains. I use the crt.sh service to monitor my domains. It provides RSS feeds (e.g. [3]) that you can hook up to your own monitoring infrastructure. I have this piped into a slack channel, so that any time a certificate is issued, I get a warning. Since I am using LE for certs, I will often cross-reference that issuance with the logs from our LE refresh bot. A future step would be to do that automatically, but this is sufficient for now. [1] https://www.certificate-transparency.org/ https://www.certificate-transparency.org/ [2] https://crt.sh https://crt.sh [3] https://crt.sh/?q=%25.ycombinator.com https://crt.sh/?q=%25.ycombinator.com

joombaga 8y ago
How do they know when a cert is issued for your domain? Edit: I never knew about this! https://tools.ietf.org/html/rfc6962 https://tools.ietf.org/html/rfc6962
- schoen 8y ago
  The certificate authorities proactively tell public logs, which these services monitor. https://www.certificate-transparency.org/ https://www.certificate-transparency.org/ https://crt.sh/monitored-logs https://crt.sh/monitored-logs
AdamGibbins 8y ago
You should also publish CAA records, to whitelist exactly what providers are allowed to issue on behalf of your domain.

whizzkid 8y ago

I would 100% use this article as my CV if I were him. Good work my friend :)

mitchtbaum 8y ago

Where's the CVE? ie. https://www.cvedetails.com/ https://www.cvedetails.com/ ...

Foxboron 8y ago
CVEs are traditionally limited to software packages. You wouldn't issue a CVE for the specification nor the cloud vendors.
- deleted 8y ago
  [deleted]
- mitchtbaum 8y ago
  "You can't improve what you don't measure.”
  - mitchtbaum 8y ago
    Rather: If you want to improve something, it helps to measure it.

johnchristopher 8y ago

Is traefik auto issuance and renewal still working ?

ohnoesjmr 8y ago

Not sure I understand how this attack works. You ask for cert for foo.com, LE tells you to provide a self-signed cert subjectAlt=foo.bar.acme.invalid. LE resolves foo.bar to some heroku endpoint as the domain lives on heroku, LE connects via TLS without verification and sends foo.bar.acme.invalid as SNI, which heroku routes to a hijacker who asked heroku to route foo bar.acme.invalid requests? Why would LE send the SNI in the first place? I thought the purpose was to prove you own the domain, not cohabit an environment where the domain is hosted? I guess it only allows hijacking certs of domains running on heroku et al envs? Also, what do host headers have to do with this? Presumably this is just a tls handshake test?

xianb 8y ago
> Why would LE send the SNI in the first place? I thought the purpose was to prove you own the domain, not cohabit an environment where the domain is hosted? The assumption was that you controlled the domain if you could return the self-signed cert with subjectAlt=foo.bar.acme.invalid when the SNI request for foo.bar.acme.invalid is made to the server your are requesting a cert for. Unfortunately the assumption didn't hold up because hosting providers shared the same routing server across domains and subdomains and those routing servers did not have controls around the subjectAlt domains used for TLS-SNI-01. > Also, what do host headers have to do with this? Presumably this is just a tls handshake test? They don't have anything to do with the weakness. It's mentioned to make the distinction that SNI is used for the cert retrieval to establish the connection and Host-header is used separately to route to the proper backend
- morpheuskafka 8y ago
  It seems that this arose from a fundamental issue with a lot of internet specifications. The person developing hosts' SNI implementation did not consider SNI to be a form of unique identification, but rather a way to establish that the server was authorized to serve it--as long as the server makes sure to serve the right files, which is a security issue for them to deal with, the certificate can be returned to anyone who requests it, even just to get a 404 page. It's like the janitor randomly trying keys from a ring, doesn't matter whose key is whose as long as it gets him in and he does what he is supposed to. The TLS-SNI-01 developers assumed, presumably based on the implementations they knew/wrote, that SNI was an identifier. When used as originally intended, of course, SNI provides the name of a real certificate desired, not a validation string, and the returned certificate authenticates a pairing of server/IP->domain not validationstring->returned certificate. I'm not sure what the TLS standard documenting SNI actually says is the right interpretation, but unless it clearly says that this might be done in the future, it seems to me like a "hackish" solution that could reasonably be expected to cause a lot of issues. This seems to be a failure of setting clear standards for critical security protocols more than anything else. By contrast, HTTPS-01 and DNS-01 operate using a known authoritative measure, ability to control the actual content of the website. If a bad actor has access to this, its game over. Only EV certs are intended to protect against this. Likewise, TLS-APLN-01 created a new protocol that could not possibly by implemented by accident or by anyone not intending to authorize issuance. It seems like nearly every major security issue (short of hard coding/crypto flaws) involves an assumption or edge case relied upon without consulting relevant standards.

hannob 8y ago

The upcoming deprecation of TLS-SNI-01 (Feb 13th) will catch a few people by surprise, because some Linux distros have been slow in reacting to this. Debian has only recently added an updated certbot to the stretch-updates repo and the normal stable repo still has an old version that defaults to TLS-SNI-01 for apache+nginx. Ubuntu 16.04, which is an LTS version, has an even older version of the letsencrypt software (before it was called certbot) and Ubunbtu doesn't seem to care.

rlpb 8y ago
> Ubuntu 16.04, which is an LTS version, has an even older version of the letsencrypt software (before it was called certbot) and Ubunbtu doesn't seem to care. Ubuntu Server developer involved with Certbot here. We do care. Updating letsencrypt/certbot in 16.04 is monumentally difficult because, from a Certbot perspective, February 2016 is prehistoric, and users of an Ubuntu stable release expect not to be regressed for all the different use cases they may have, not all of which we even know about. Updating a stable release is difficult for these reasons normally. Now add in the complication of five different interacting source packages, an upstream project rename, and the need to not regress the behaviour of a key library which must be updated but users may be depending on directly, and hopefully you can see the difficulty of this task. We are working on it though, and you can follow progress here: https://launchpad.net/bugs/1640978 https://launchpad.net/bugs/1640978. I'm still hopeful that this will land in time. In the meantime, if you can't wait, are using 16.04 and must have it now, then you have a number of options: 1) Use certbot-auto as recommended by upstream. 2) Use 18.04, which isn't affected as it shipped with a new-enough certbot package. 3) Try the (experimental) snap: https://forum.snapcraft.io/t/call-for-testing-certbot-lets-encrypt/7990 https://forum.snapcraft.io/t/call-for-testing-certbot-lets-e... (though I don't recommend this for production). Relative to the deb packaging, the snap has been trivial to develop and maintain and the edge channel keeps up with upstream master automatically as long as CI passes (and keeping it passing has been a relative breeze).

walrus01 8y ago

It is worth noting that the version of 'certbot' currently packaged for debian-stable (stretch) is very old, and attempts to use tls-sni-01 as the default method for verification. Strongly recommended to enable stretch-backports and update certbot to v0.28.

joering2 8y ago

So thru my network I see 278 servers I need to patch. This will take forever, or couple days of work. Next time I will simply spend few bucks on an annual certificate instead.

Dylan16807 8y ago
A few bucks, or a few bucks multiplied by 278? If you'd be replacing everything with a wildcard because it's easier, you can get a free wildcard from Let's Encrypt too.
GuyPostington 8y ago
Or you know, update your packages.

B-Con 8y ago

Rewording (and hopefully simplifying) for my own benefit because I think the OP is a little loose with the details. There are two parts: Part 1: Subdomain takeover 1) DNS for domain X points to service Y (Y=Heroku in this case). 2) Service Y allows anyone to claim an unused domain. 3) Domain X is then claimed by an attacker. 4) Domain X is then served by an attacker. 5) Bonus points if the owner of X has a wildcarded HTTPS cert, so the hijacked subdomain gets HTTPS automatically. So now attacker serves content on a domain they don't own (possibly over HTTPS). This is already a known cloud hosting mis-configuration problem. In this case, investor.example.com for HTTPS was not claimed on Heroku but DNS was pointed to Heroku, so it was vulnerable to subdomain takeover. However, the author only had HTTPS access and example.com did NOT have wildcarded cert, so the author was in an edge case of having control of a subdomain but only over HTTPS with no cert. This prompted them to look for... Part 2: Abusing TLS-SNI-01 validation TLS-SNI-01 works by having LetsEncrypt request a cert with a secret from the specified domain. 1) LE does a DNS lookup of the desired domain. 2) LE connects to that domain using TLS-SNI. SNI specifies a hostname in the pre-encryption request. 3) ACME TLS-SNI-01 has the SNI requested domain in the form foo.bar.acme.invalid. 4) Author claimed the necessary foo.bar.acme.invalid hostname on Heroku (and later AWS). 5) Author initiated a LetsEncrypt cert request for a different domain on the same hosting provider. 6) When LE made the SNI request to that domain it did a DNS lookup and got pointed to the shared hosting provider, but the SNI hostname was of the form foo.bar.acme.invalid which was controlled by the attacker, so it was passed to the attacker's app. 6.a (Side note: Because there was no valid cert offered by the hosting provider, I'm guessing the front-ends passed the SNI request through to the backend app. Because if there was a cert configured for the Heroku app then I think the front-ends just serve the cert directly without checking with the backend app.) 7) The attacker then finished the LE cert negotiation and got a cert on behalf of the other domain. I'm a little fuzzy on exactly how choosing the domain in (3) and (6) works since the spec[0] doesn't seem to match what the blog post shows, but I'm sure it works somehow. Corrections/clarifications welcome. Related specs: [0] https://tools.ietf.org/html/draft-ietf-acme-acme-01#page-40 https://tools.ietf.org/html/draft-ietf-acme-acme-01#page-40 [1] https://tools.ietf.org/html/rfc6066#page-6 https://tools.ietf.org/html/rfc6066#page-6

kpcyrd 8y ago

The title on hackernews is highly misleading, there's quite a difference between "for any domain" and "for any domain using shared hosting".