10 ms·

Issue with TLS-SNI-01 and Shared Hosting Infrastructure

160 points by jlgaddis 9y ago

jlgaddis 9y ago
> At this time, we believe that the issue can be addressed by having certain services providers implement stronger controls for domains hosted on their infrastructure. Requiring unrelated third-parties to take specific action(s) to prevent Let's Encrypt from incorrectly issuing certificates doesn't sound like a valid solution to this issue. What about the shared hosting providers that are "affected" that decide to do nothing to "fix" this?
- ZiiS 9y ago
  Slightly mitigated by only effecting the domains of people who chose to use such a hosting provider; but I agree this doesn't seem to meet the bar we should expect from a CA.
- jmhodges 9y ago
  There are equivalent attacks on both the DNS and HTTP challenges as described in the Baseline Requirements and ACME, and those are expected to be mitigated (and are) by hosting providers. Let's Encrypt isn't bound by the Baseline Requirements to mitigate this for these hosting providers, but is doing so out of a esprit-de-corps while the web figures out how it wants to handle this. Hosting providers haven't, previously, thought of TLS serving as an individual user service before (though the Baseline Requirements do) and are having to work out the kinks of that.
  - jo909 9y ago
    I don't think they are really equivalent. We are talking about Domain Validated Certificates, and for DNS and HTTP "attacks" you need to be able to serve content in the name/as part of that domain. In this attack you need to be able to serve content in the "name of" a made up TLS name under an IP you share with the domain you attack (which is very common).
    - jmhodges 9y ago
      My "Hosting providers haven't, previously, thought of TLS serving as an individual user service before (though the Baseline Requirements do) and are having to work out the kinks of that." covers what you're trying to get at here.
      - jo909 9y ago
        I agree completely with that sentence factually and as an explanation why we have such issues, but still think HTTP and DNS are fundamentally different to this issue with the TLS-SNI challenge. The domain under attack is not part of the actual challenge process here. As a hosting provider I never see it and it plays no role in the decision of what information I reply with. At no point do I serve content under the "wrong" domain. At no point does the attacker show any control over the domain being validated.
        jmhodges 9y ago
        Well, no, the DNS resolved to these endpoints, just like in an HTTP attack, and then this is as if the Host header wasn't checked by the HTTP provider. You and these few specific hosting providers didn't think they had to look at the SANs, but, in fact, the Baseline Requirements expected them to verify that the SANs in the certs are controlled by the same user. There's a reasonable disagreement but I (and others[1]) liken this the "postmaster@" attacks. At some point, for every protocol the hosting provider handles, we always end up having them do a bit more work then they thought they had to do but them's the breaks when dealing with the modern internet. [1] https://twitter.com/sleevi_/status/951041801368035328 https://twitter.com/sleevi_/status/951041801368035328
        jo909 9y ago
        I don't advocate for such hosting providers to not mitigate that attack. It's a real problem and needs a real solution, no matter the technical or political reasons that lead to this. Those hosting providers might not ever want to use LE for their customers and might arguably not be "at fault", but still their customers are at risk and they should take steps to protect them. But I still think it's a different problem in this case. In the end I suppose my argument is that this is a design flaw in this challenge and we ideally should not use it in it's current form, just as we no longer should use postmaster@ for domain validation (but the technical argument against postmaster@ is again a fundamentally different one). Edit: I realized I was wrong and removed one part of my response regarding IP lookup as a positive sign of domain ownership.
- tialaramex 9y ago
  In specific answer to your second question, Let's Encrypt intends to blacklist IPs for those providers from using tls-sni-01 proof of control. Now, requiring unrelated third-parties to take action is (perhaps unfortunately) nothing new for DV. Back in the day lots of the web mail providers got bitten because some CA would decide that, say, receipt of email to ssl-certificate-officer@example.com is proof of control of example.com, so you just sign up for web mail and say you want the email address ssl-certificate-officer and bingo you get a certificate for a few bucks. This was exploited over and over. Even today, the Ten Blessed Methods let a CA choose any email address you listed in WHOIS (fair enough, don't list anybody else's address in WHOIS) _or_ any of 'admin', 'administrator', 'webmaster', 'hostmaster', or 'postmaster'. So when you build a web mail system you need to explicitly blacklist those five email addresses even if you would never need an account named "webmaster" or "admin", if you fail to take this specific action anybody can get certificates for your domain by talking to any of the commercial CAs that use a method complying with 3.2.2.4.4. Let's Encrypt already disabled a feature of http-01 validation before going live to avoid needing bulk web hosts and CDNs to make a sensible configuration choice, because surveys suggested that lots of them had left insecure default choices enabled. Or another email example (again thus not affecting Let's Encrypt): Lots of companies purchased anti-phishing systems that scan all incoming email to check for dodgy links. These systems typically dereference all the links, check what they find and quarantine suspicious emails. Several CAs had systems which sent an email saying "Just click here to authorize issuance". The anti-phishing system follows the link, now it's clicked, anyone could get a certificate for these domains because the anti-phishing system would "authorize" anything they requested... Now, all these are historical rather than current examples, but they do illustrate that "requiring unrelated parties to take specific actions" won't be new or unique to Let's Encrypt.
  - teddyh 9y ago
    > So when you build a web mail system you need to explicitly blacklist those five email addresses… Actually, you should blacklist all of the addresses listed in RFC 2142¹. Also consider this blog post for additional addresses to block: https://ldpreload.com/blog/names-to-reserve https://ldpreload.com/blog/names-to-reserve 1. https://tools.ietf.org/html/rfc2142 https://tools.ietf.org/html/rfc2142
- y0ghur7_xxx 9y ago
  > What about the shared hosting providers that are "affected" that decide to do nothing to "fix" this? The problem is that I can upload a self signed certificate on a shared host for a domain I don't own. This, in and for itself, should not be a problem. I did nothing wrong, and the shared host did nothing wrong as well: it only allowed me to upload a self signed cert for an inexistent domain. IMHO, those two actions should not allow someone to get a valid cert for a domain they don't control, and whose only fault is that they pointed their A record to the shared domain host. I don't think TLS-SNI is sufficient proof of Domain Control. IMHO it should be sacked.
  - lathiat 9y ago
    As someone with a long history in shared hosting, I have to agree. Basically requesting that ".acme.invalid" SNI proves ownership only of the ".acme.invalid" name and not the actual domain name in any way. I don't see any way this scheme can work. The only way I can see this working is having to pre-install a self-signed certificate with the same private key as the requested certificate to sign, on the actual domain name. I think that can work, can't it? I'd have to try it with openssl.. They may be able to fix it for larger providers but you'll forever have many smaller providers that will never mitigate the issue. The obvious problem with that is temporarily killing the real SSL certificate during initial setup. Though in theory they could re-use the private key.
  - Ajedi32 9y ago
    > the shared host did nothing wrong as well I'm not so sure about that. Is it really a good idea to let customers configure security settings (e.g. the TLS cert used) for domains they don't own? I suppose as long as they're only doing it for invalid domains it shouldn't be an issue, but what if a customer were to set a cert for sitea.example (which they don't own) and then later the real owner of sitea.example comes along and wants to set up their site on the same host? Just doesn't seem like good practice to me. Maybe the solution here is to make the domain name sent by the SNI challenge a subdomain of the domain being validated? If a shared hosting site allows customers to set TLS certs for subdomains of a _completely different customer's_ domain name, that's most definitely a security issue regardless of whether or not TLS-SNI validation exists.
benmmurphy 9y ago
does this mean you can collide legitimate certificates on these hosting providers? like someone is hosting www.foo.com can I upload a www.foo.com certificate then it will choose which cert to use based on some random factors. or maybe it only works on names that haven't been uploaded so i can block someone from uploading certificates for a future domain they might want to use.
- tscs37 9y ago
  I'd say no, though it depends on the provider. Afaict the vulnerability is simply "users can upload arbitrary certs and have the server use them based on SNI", which does have some legitimate use cases (private CA)
- jo909 9y ago
  Maybe, but that is a different issue. If as a hosting provider I have accounts for foo.com and bar.com, I certainly need to make sure to never reply to requests for foo.com with data from the account bar.com. But I might still allow the account bar.com to upload a certificate that includes bar.com www.bar.com bar.io getbar.com or any other domain name even if those are not registered in my platform. So what if bar.io is actually not owned by the same person as bar.com? The DNS for bar.io does not point to my platform, no real user will connect to me. If an attacker is also able to change the DNS for bar.io (say by an MITM attack in a public wifi) it is of no real consequence that he uses my platform. He could just as well respond with an IP completely of his own and do everything himself. That is why we have SSL in the first place.
  - benmmurphy 9y ago
    so if someone has www.foo.com registered with a provider (but has wildcard DNS pointing *.foo.com to the provider) would it be possible for someone who has another domain bar.com to upload a cert that would be used for x.foo.com. because even though no content would be served from bar.com that is kind of a weird situation. like i think the way ACME tls-sni works is broken because it should be using the DNS hierarchy to make the SNI request [maybe something like: 773c7d.13445a.acme.foo.com] but maybe some providers are still broken even with this fix because they let people upload certs with names that belong to other clients.
paralelogram 9y ago
I can create "773c7d.13445a.acme.invalid" in almost all shared hosting control panels I have access to. When I send an HTTP request with Host: 773c7d.13445a.acme.invalid, the server responds with a file from ~/domains/773c7d.13445a.acme.invalid/public_html or a similar directory available through my FTP account. When I connect using openssl s_client ... -servername 773c7d.13445a.acme.invalid, the server sends a certificate configured for 773c7d.13445a.acme.invalid in my control panel. Is this a problem for Let's Encrypt? Doesn't Let's Encrypt's verification require creating files with random names in http://example.com/.well-known/acme-challenge http://example.com/.well-known/acme-challenge where example.com is the certificate's common name?
- pfg 9y ago
  > Is this a problem for Let's Encrypt? Doesn't Let's Encrypt's verification require creating files with random names in http://example.com/.well-known/acme-challenge http://example.com/.well-known/acme-challenge where example.com is the certificate's common name? That applies to the http-01 challenge. The tls-sni-01 challenge works solely based on the returned certificate. If the SAN value in the certificate matches the SNI value sent by the validation server, the challenge succeeds. Would you mind sharing which control panel you tested this with?
  - paralelogram 9y ago
    DirectAdmin, the most popular webhosting control panel in my country. In my opinion this is not a bug because when I need to test a website, I often create an invalid hostname on the server and add the server's IP address to my computer's /etc/hosts. When I need HTTPS, I upload a certificate for the test hostname signed by my private CA.
    - pfg 9y ago
      Thanks. I signed up for the first shared web hosting provider I could find that uses DirectAdmin and was able to reproduce this. I'll bring this up in the relevant thread on mozilla.dev.security.policy, this is definitely concerning.
jwilk 9y ago
See also: https://news.ycombinator.com/item?id=16112237 https://news.ycombinator.com/item?id=16112237
mh-cx 9y ago
Does this basically say, that letsencrypt will no longer work in a virtual hosting setup? What about existing certs? Can they at least be renewed when they expire?
- luto 9y ago
  Let's Encrypt works just fine in a virtual hosting setup using the http-01 or dns-01 challenges. In fact, all (shared/virtual hosting) users in my company automatically get certificates this way.
  - v4tab 9y ago
    Is there a way to tell whether I am using http-01, dns-01 or TLS-SNI-01? I'm using the letsencrypt nginx module.
    - discreditable 9y ago
      Seems to use tls-sni-01. This is my guess by looking at the certbot-nginx source directory: https://github.com/certbot/certbot/tree/b1826d657ffc7c278041cc0c8e14dc2ba23f95e3/certbot-nginx/certbot_nginx https://github.com/certbot/certbot/tree/b1826d657ffc7c278041...
      - mnordhoff 9y ago
        That's correct. Certbot can use different plugins for validating the name and for installing the certificate. You can configure HTTP-01 to work and use "certbot -a webroot -i nginx -w /path/to/whatever -d example.com -d www.example.com". https://community.letsencrypt.org/t/solution-client-with-the-currently-selected-authenticator-does-not-support-any-combination-of-challenges-that-will-satisfy-the-ca/49983 https://community.letsencrypt.org/t/solution-client-with-the...
jcassee 9y ago
[Copying my comment from https://news.ycombinator.com/item?id=16112237 https://news.ycombinator.com/item?id=16112237] Unfortunately, this means that Traefik's default Let's Encrypt integration (without setting a DNS provider) does not work anymore. Although the logs now say "could not find solver for: http-01", they actually use tls-sni-01.
- BillinghamJ 9y ago
  The entire Golang autocert functionality relies on this - which presumably Traefik is using
  - fapjacks 9y ago
    Through a twist of fate, I was looking into this over the weekend, and you're correct about Traefik.
rgbrenner 9y ago
Over the next 48 hours we will be building a list of vulnerable providers and their associated IP addresses. Our tentative plan, once the list is completed, is to re-enable the TLS-SNI-01 challenge type with vulnerable providers blocked from using it. This is a terrible solution. You're assuming you can scour the internet and find every shared hosting provider affected, and add them to a list. And then you're also going to keep that list updated. That's crazy.. an impossible task. So after this is done, Let's Encrypt will continue issuing certificates improperly on any shared hosting providers that didn't get added to this list. Attackers just need to find a provider and check a list now. I'm actually kind of appalled that this is what you've come up with.
- deleted 9y ago
  [deleted]
- mdhardeman 9y ago
  I concur in full with your commentary here. LetsEncrypt, through thoughtful planning and action -- even when it was inconvenient for them -- has established themselves as a shining beacon of best practice. For better or worse, when you become the best practice example, you are held by the world -- and especially by your peers -- to a higher standard. I can not imagine how they would see a return to production status of TLS-SNI-01 as anything other than painting a target on themselves for their various detractors to take aim at. It is no secret that numerous other parties in the CA space would love to see them slip and fall.
- DanielDent 9y ago
  I think this issue can be described as a form of confused deputy problem. Is not the real solution to have the confused deputies stop acting in a confused manner? If this issue were confined to a handful of small hosting companies almost no one uses, then I don't think this would be described as a problem with Let's Encrypt - it would be described as a problem with those hosting companies. The only party that can truly fix the issue of the incorrect user being granted access (directly or indirectly) to a domain pointed at shared infrastructure is the operator of that infrastructure.
BillinghamJ 9y ago
Ultimately, if you're attempting to prove hostname control, it would seem fundamentally incorrect to make a request with a Host header or SNI hostname set to anything other than the hostname requested for issuance. I don't really understand why this didn't come up while considering the protocol initially. Why aren't we using ALPN for this? (ALPN is used to negotiate HTTP/2 without prior knowledge of the server's capability - https://en.wikipedia.org/wiki/Application-Layer_Protocol_Negotiation https://en.wikipedia.org/wiki/Application-Layer_Protocol_Neg...)
- rgbrenner 9y ago
  Yes... as long as SNI is used and valid on the internet... it is incorrect to assume IP = domain control.. which is the assumption that the TLS-SNI-01/02 challenges are based on. These challenges are fatally flawed in their current form, and should be discontinued. (Your solution in the first sentence changes it to domain control = domain control. Obviously correct.)
- tonyarkles 9y ago
  > it would seem fundamentally incorrect to make a request with a Host header or SNI hostname set to anything other than the hostname requested for issuance. Pondering why they'd do it the way it is right now, my best guess would be that it allows you to get a LetsEncrypt certificate while still having a valid certificate from another CA. For example, example.com currently has a Symantec certificate that is soon going to be invalidated, and I want to swap over to an LE cert with 0 downtime.
  - BillinghamJ 9y ago
    You could still do that! Just accept LE’s requests with the existing Symantec cert and respond as needed. Or use the DNS challenge. Or use something like ALPN instead.
    - tonyarkles 9y ago
      How would LE do SNI validation using the Symantec cert though? I can't add anything to it without invalidating it. My understanding is that that's why they do the .acme.invalid SNI responses; so that the server can respond with a self-signed cert that's not going to interfere with anything else. DNS and ALPN are both great solution, but here's why neither of them work all that well in my specific circumstances, but TLS-SNI works awesome. - I can provision VMs at will, but I don't personally have access to DNS. To get a DNS record added, I need to submit a change request. I haven't done the DNS challenge, and I'm not sure if a new TXT record gets generated when you renew a cert or if it's static after initial creation, but either way, DNS very much adds a painful manual step to cert provisioning here. - ALPN would be great, if it were baked into the default Centos Apache :). That, I suspect, would be a long time coming... Right now I generally just use HTTP validation, because I've worked out a pattern that generally works pretty good. I set up an Apache rule that auto-redirects all traffic from http to https anyway; I just add a condition to it so that /.well-known/ gets served off the filesystem before redirecting everything else.
rgbrenner 9y ago
How does TLS-SNI-02 fix this issue? ... Edit: I've removed most of my post here... This was already acknowledged in the article. 02 is vulnerable also.
- mdhardeman 9y ago
  I think the only fix is to create TLS-SNI-03 in which the only dnsName component in the self-signed certificate is a well known child of the domain label to be validated. Validating www.abc.com, SNI and dnsName is well-known-acme-pki.www.abc.com and the certificate should have some other parameter stuffed with a challenge response that is defined to be a challenge response calculated over the inputs of a random token provided to the requestor by the CA and the account key of the requestor.
mappu 9y ago
This is not specifically a vulnerability in TLS-SNI-01 - more like a major misconfiguration by the shared hosting provider, to share ACME secrets. I guess TLS-SNI-01 does share part of the blame for making this possible in the first place. It would be great if TLS-SNI-01 could be salvaged, for all the binaries out there - but i don't think a blacklist is a great long-term solution. How about a strict one IP = one domain lockdown until you can prove that you don't share challenges? i.e. negative responses. EDIT: See reply
- rgbrenner 9y ago
  I disagree. It's a fatal flaw in TLS-SNI-01/02. Here's why: All of the information required to solve the challenge is in the attackers control. SAN-A and SAN-B (in v02) are known by the attacker. And no action or information from the target domain is required to solve the challenge. Every shared hosting provider and CDN that lets you pick a domain and upload a custom certificate can be used to solve the challenge. Your solution of restricting it to ip = one domain just reduces the vulnerable sites to sites that don't use LE and are on a shared provider. (Also note, if the provider does not implement LE.. then they are vulnerable to this attack against all of the customers.. with your solution: 1 customer for each IP owned by the service) As long as SNI is part of the TLS standard, it is incorrect to assume IP = domain ownership.
  - mappu 9y ago
    You're right. Thanks for clarifying.
tialaramex 9y ago
My _guess_ at this point is that there's a bunch of scary corner cases with the behaviour that makes this possible so that if your bulk host or CDN does this you need to be _very_ careful even if Let's Encrypt abolishes tls-sni family proofs of control. If I'm right, fixing this is pretty urgent for anyone with untrusted users sharing an IP address. It looks very much as if affected hosts/CDNs are letting any customer impersonate any other customer with the only caveats being that you've got to get in first and you need the victim customer to make the config change that causes this to be useful. Suppose I am victim.example, I have a contract with Funky Jim's Hosting, and I have configured *.clowns.victim.example to go to a server at Funky Jim's but I've only every bothered setting up www.clowns.victim.example, images.clown.victim.example and js.clowns.victim.example in Funky Jim's config system. I use HTTPS, so the valuable Clown Login tokens available on www.clowns.victim.example are protected - aren't they? Some loser, let's call her Sad Carol buys a basic account from Funky Jim, and tells Funky Jim they want to add a new site, busted.clowns.victim.example. Funky Jim OKs it, because that's "harmless" right? No need for Carol to prove to Jim that she owns it, if not why would it matter? But wait, actually busted.clowns.victim.example matches my wildcard DNS entry, it goes to the right server and serves up Carol's site - now Sad Carol can get all my bloody Clown Login cookies sent to "her" web site just by injecting "her" site into an ad inventory or whatever.
- mdhardeman 9y ago
  An excellent example of a plausible attack scenario. Having said that, I think what we're all dying to know is what was it, initially, that made Carol so sad? And, indeed, was it that same sadness which stagnated, fermented, and evolved into the obvious hatred and antisocial behaviors that she exhibits here?
nodesocket 9y ago
> At this time, we believe that the issue can be addressed by having certain services providers implement stronger controls for domains hosted on their infrastructure. Is DigitalOcean one of these service providers that need to implement stronger controls?

160 points by jlgaddis 9y ago

> At this time, we believe that the issue can be addressed by having certain services providers implement stronger controls for domains hosted on their infrastructure. Requiring unrelated third-parties to take specific action(s) to prevent Let's Encrypt from incorrectly issuing certificates doesn't sound like a valid solution to this issue. What about the shared hosting providers that are "affected" that decide to do nothing to "fix" this?

ZiiS 9y ago
Slightly mitigated by only effecting the domains of people who chose to use such a hosting provider; but I agree this doesn't seem to meet the bar we should expect from a CA.
jmhodges 9y ago
There are equivalent attacks on both the DNS and HTTP challenges as described in the Baseline Requirements and ACME, and those are expected to be mitigated (and are) by hosting providers. Let's Encrypt isn't bound by the Baseline Requirements to mitigate this for these hosting providers, but is doing so out of a esprit-de-corps while the web figures out how it wants to handle this. Hosting providers haven't, previously, thought of TLS serving as an individual user service before (though the Baseline Requirements do) and are having to work out the kinks of that.
- jo909 9y ago
  I don't think they are really equivalent. We are talking about Domain Validated Certificates, and for DNS and HTTP "attacks" you need to be able to serve content in the name/as part of that domain. In this attack you need to be able to serve content in the "name of" a made up TLS name under an IP you share with the domain you attack (which is very common).
  - jmhodges 9y ago
    My "Hosting providers haven't, previously, thought of TLS serving as an individual user service before (though the Baseline Requirements do) and are having to work out the kinks of that." covers what you're trying to get at here.
    - jo909 9y ago
      I agree completely with that sentence factually and as an explanation why we have such issues, but still think HTTP and DNS are fundamentally different to this issue with the TLS-SNI challenge. The domain under attack is not part of the actual challenge process here. As a hosting provider I never see it and it plays no role in the decision of what information I reply with. At no point do I serve content under the "wrong" domain. At no point does the attacker show any control over the domain being validated.
      - jmhodges 9y ago
        Well, no, the DNS resolved to these endpoints, just like in an HTTP attack, and then this is as if the Host header wasn't checked by the HTTP provider. You and these few specific hosting providers didn't think they had to look at the SANs, but, in fact, the Baseline Requirements expected them to verify that the SANs in the certs are controlled by the same user. There's a reasonable disagreement but I (and others[1]) liken this the "postmaster@" attacks. At some point, for every protocol the hosting provider handles, we always end up having them do a bit more work then they thought they had to do but them's the breaks when dealing with the modern internet. [1] https://twitter.com/sleevi_/status/951041801368035328 https://twitter.com/sleevi_/status/951041801368035328
        jo909 9y ago
        I don't advocate for such hosting providers to not mitigate that attack. It's a real problem and needs a real solution, no matter the technical or political reasons that lead to this. Those hosting providers might not ever want to use LE for their customers and might arguably not be "at fault", but still their customers are at risk and they should take steps to protect them. But I still think it's a different problem in this case. In the end I suppose my argument is that this is a design flaw in this challenge and we ideally should not use it in it's current form, just as we no longer should use postmaster@ for domain validation (but the technical argument against postmaster@ is again a fundamentally different one). Edit: I realized I was wrong and removed one part of my response regarding IP lookup as a positive sign of domain ownership.
tialaramex 9y ago
In specific answer to your second question, Let's Encrypt intends to blacklist IPs for those providers from using tls-sni-01 proof of control. Now, requiring unrelated third-parties to take action is (perhaps unfortunately) nothing new for DV. Back in the day lots of the web mail providers got bitten because some CA would decide that, say, receipt of email to ssl-certificate-officer@example.com is proof of control of example.com, so you just sign up for web mail and say you want the email address ssl-certificate-officer and bingo you get a certificate for a few bucks. This was exploited over and over. Even today, the Ten Blessed Methods let a CA choose any email address you listed in WHOIS (fair enough, don't list anybody else's address in WHOIS) _or_ any of 'admin', 'administrator', 'webmaster', 'hostmaster', or 'postmaster'. So when you build a web mail system you need to explicitly blacklist those five email addresses even if you would never need an account named "webmaster" or "admin", if you fail to take this specific action anybody can get certificates for your domain by talking to any of the commercial CAs that use a method complying with 3.2.2.4.4. Let's Encrypt already disabled a feature of http-01 validation before going live to avoid needing bulk web hosts and CDNs to make a sensible configuration choice, because surveys suggested that lots of them had left insecure default choices enabled. Or another email example (again thus not affecting Let's Encrypt): Lots of companies purchased anti-phishing systems that scan all incoming email to check for dodgy links. These systems typically dereference all the links, check what they find and quarantine suspicious emails. Several CAs had systems which sent an email saying "Just click here to authorize issuance". The anti-phishing system follows the link, now it's clicked, anyone could get a certificate for these domains because the anti-phishing system would "authorize" anything they requested... Now, all these are historical rather than current examples, but they do illustrate that "requiring unrelated parties to take specific actions" won't be new or unique to Let's Encrypt.
- teddyh 9y ago
  > So when you build a web mail system you need to explicitly blacklist those five email addresses… Actually, you should blacklist all of the addresses listed in RFC 2142¹. Also consider this blog post for additional addresses to block: https://ldpreload.com/blog/names-to-reserve https://ldpreload.com/blog/names-to-reserve 1. https://tools.ietf.org/html/rfc2142 https://tools.ietf.org/html/rfc2142
y0ghur7_xxx 9y ago
> What about the shared hosting providers that are "affected" that decide to do nothing to "fix" this? The problem is that I can upload a self signed certificate on a shared host for a domain I don't own. This, in and for itself, should not be a problem. I did nothing wrong, and the shared host did nothing wrong as well: it only allowed me to upload a self signed cert for an inexistent domain. IMHO, those two actions should not allow someone to get a valid cert for a domain they don't control, and whose only fault is that they pointed their A record to the shared domain host. I don't think TLS-SNI is sufficient proof of Domain Control. IMHO it should be sacked.
- lathiat 9y ago
  As someone with a long history in shared hosting, I have to agree. Basically requesting that ".acme.invalid" SNI proves ownership only of the ".acme.invalid" name and not the actual domain name in any way. I don't see any way this scheme can work. The only way I can see this working is having to pre-install a self-signed certificate with the same private key as the requested certificate to sign, on the actual domain name. I think that can work, can't it? I'd have to try it with openssl.. They may be able to fix it for larger providers but you'll forever have many smaller providers that will never mitigate the issue. The obvious problem with that is temporarily killing the real SSL certificate during initial setup. Though in theory they could re-use the private key.
- Ajedi32 9y ago
  > the shared host did nothing wrong as well I'm not so sure about that. Is it really a good idea to let customers configure security settings (e.g. the TLS cert used) for domains they don't own? I suppose as long as they're only doing it for invalid domains it shouldn't be an issue, but what if a customer were to set a cert for sitea.example (which they don't own) and then later the real owner of sitea.example comes along and wants to set up their site on the same host? Just doesn't seem like good practice to me. Maybe the solution here is to make the domain name sent by the SNI challenge a subdomain of the domain being validated? If a shared hosting site allows customers to set TLS certs for subdomains of a _completely different customer's_ domain name, that's most definitely a security issue regardless of whether or not TLS-SNI validation exists.

benmmurphy 9y ago

does this mean you can collide legitimate certificates on these hosting providers? like someone is hosting www.foo.com can I upload a www.foo.com certificate then it will choose which cert to use based on some random factors. or maybe it only works on names that haven't been uploaded so i can block someone from uploading certificates for a future domain they might want to use.

tscs37 9y ago
I'd say no, though it depends on the provider. Afaict the vulnerability is simply "users can upload arbitrary certs and have the server use them based on SNI", which does have some legitimate use cases (private CA)
jo909 9y ago
Maybe, but that is a different issue. If as a hosting provider I have accounts for foo.com and bar.com, I certainly need to make sure to never reply to requests for foo.com with data from the account bar.com. But I might still allow the account bar.com to upload a certificate that includes bar.com www.bar.com bar.io getbar.com or any other domain name even if those are not registered in my platform. So what if bar.io is actually not owned by the same person as bar.com? The DNS for bar.io does not point to my platform, no real user will connect to me. If an attacker is also able to change the DNS for bar.io (say by an MITM attack in a public wifi) it is of no real consequence that he uses my platform. He could just as well respond with an IP completely of his own and do everything himself. That is why we have SSL in the first place.
- benmmurphy 9y ago
  so if someone has www.foo.com registered with a provider (but has wildcard DNS pointing *.foo.com to the provider) would it be possible for someone who has another domain bar.com to upload a cert that would be used for x.foo.com. because even though no content would be served from bar.com that is kind of a weird situation. like i think the way ACME tls-sni works is broken because it should be using the DNS hierarchy to make the SNI request [maybe something like: 773c7d.13445a.acme.foo.com] but maybe some providers are still broken even with this fix because they let people upload certs with names that belong to other clients.

paralelogram 9y ago

I can create "773c7d.13445a.acme.invalid" in almost all shared hosting control panels I have access to. When I send an HTTP request with Host: 773c7d.13445a.acme.invalid, the server responds with a file from ~/domains/773c7d.13445a.acme.invalid/public_html or a similar directory available through my FTP account. When I connect using openssl s_client ... -servername 773c7d.13445a.acme.invalid, the server sends a certificate configured for 773c7d.13445a.acme.invalid in my control panel. Is this a problem for Let's Encrypt? Doesn't Let's Encrypt's verification require creating files with random names in http://example.com/.well-known/acme-challenge http://example.com/.well-known/acme-challenge where example.com is the certificate's common name?

pfg 9y ago
> Is this a problem for Let's Encrypt? Doesn't Let's Encrypt's verification require creating files with random names in http://example.com/.well-known/acme-challenge http://example.com/.well-known/acme-challenge where example.com is the certificate's common name? That applies to the http-01 challenge. The tls-sni-01 challenge works solely based on the returned certificate. If the SAN value in the certificate matches the SNI value sent by the validation server, the challenge succeeds. Would you mind sharing which control panel you tested this with?
- paralelogram 9y ago
  DirectAdmin, the most popular webhosting control panel in my country. In my opinion this is not a bug because when I need to test a website, I often create an invalid hostname on the server and add the server's IP address to my computer's /etc/hosts. When I need HTTPS, I upload a certificate for the test hostname signed by my private CA.
  - pfg 9y ago
    Thanks. I signed up for the first shared web hosting provider I could find that uses DirectAdmin and was able to reproduce this. I'll bring this up in the relevant thread on mozilla.dev.security.policy, this is definitely concerning.

jwilk 9y ago

See also: https://news.ycombinator.com/item?id=16112237 https://news.ycombinator.com/item?id=16112237

mh-cx 9y ago

Does this basically say, that letsencrypt will no longer work in a virtual hosting setup? What about existing certs? Can they at least be renewed when they expire?

luto 9y ago
Let's Encrypt works just fine in a virtual hosting setup using the http-01 or dns-01 challenges. In fact, all (shared/virtual hosting) users in my company automatically get certificates this way.
- v4tab 9y ago
  Is there a way to tell whether I am using http-01, dns-01 or TLS-SNI-01? I'm using the letsencrypt nginx module.
  - discreditable 9y ago
    Seems to use tls-sni-01. This is my guess by looking at the certbot-nginx source directory: https://github.com/certbot/certbot/tree/b1826d657ffc7c278041cc0c8e14dc2ba23f95e3/certbot-nginx/certbot_nginx https://github.com/certbot/certbot/tree/b1826d657ffc7c278041...
    - mnordhoff 9y ago
      That's correct. Certbot can use different plugins for validating the name and for installing the certificate. You can configure HTTP-01 to work and use "certbot -a webroot -i nginx -w /path/to/whatever -d example.com -d www.example.com". https://community.letsencrypt.org/t/solution-client-with-the-currently-selected-authenticator-does-not-support-any-combination-of-challenges-that-will-satisfy-the-ca/49983 https://community.letsencrypt.org/t/solution-client-with-the...

jcassee 9y ago

[Copying my comment from https://news.ycombinator.com/item?id=16112237 https://news.ycombinator.com/item?id=16112237] Unfortunately, this means that Traefik's default Let's Encrypt integration (without setting a DNS provider) does not work anymore. Although the logs now say "could not find solver for: http-01", they actually use tls-sni-01.

BillinghamJ 9y ago
The entire Golang autocert functionality relies on this - which presumably Traefik is using
- fapjacks 9y ago
  Through a twist of fate, I was looking into this over the weekend, and you're correct about Traefik.

rgbrenner 9y ago

Over the next 48 hours we will be building a list of vulnerable providers and their associated IP addresses. Our tentative plan, once the list is completed, is to re-enable the TLS-SNI-01 challenge type with vulnerable providers blocked from using it. This is a terrible solution. You're assuming you can scour the internet and find every shared hosting provider affected, and add them to a list. And then you're also going to keep that list updated. That's crazy.. an impossible task. So after this is done, Let's Encrypt will continue issuing certificates improperly on any shared hosting providers that didn't get added to this list. Attackers just need to find a provider and check a list now. I'm actually kind of appalled that this is what you've come up with.

deleted 9y ago
[deleted]
mdhardeman 9y ago
I concur in full with your commentary here. LetsEncrypt, through thoughtful planning and action -- even when it was inconvenient for them -- has established themselves as a shining beacon of best practice. For better or worse, when you become the best practice example, you are held by the world -- and especially by your peers -- to a higher standard. I can not imagine how they would see a return to production status of TLS-SNI-01 as anything other than painting a target on themselves for their various detractors to take aim at. It is no secret that numerous other parties in the CA space would love to see them slip and fall.
DanielDent 9y ago
I think this issue can be described as a form of confused deputy problem. Is not the real solution to have the confused deputies stop acting in a confused manner? If this issue were confined to a handful of small hosting companies almost no one uses, then I don't think this would be described as a problem with Let's Encrypt - it would be described as a problem with those hosting companies. The only party that can truly fix the issue of the incorrect user being granted access (directly or indirectly) to a domain pointed at shared infrastructure is the operator of that infrastructure.

BillinghamJ 9y ago

Ultimately, if you're attempting to prove hostname control, it would seem fundamentally incorrect to make a request with a Host header or SNI hostname set to anything other than the hostname requested for issuance. I don't really understand why this didn't come up while considering the protocol initially. Why aren't we using ALPN for this? (ALPN is used to negotiate HTTP/2 without prior knowledge of the server's capability - https://en.wikipedia.org/wiki/Application-Layer_Protocol_Negotiation https://en.wikipedia.org/wiki/Application-Layer_Protocol_Neg...)

rgbrenner 9y ago
Yes... as long as SNI is used and valid on the internet... it is incorrect to assume IP = domain control.. which is the assumption that the TLS-SNI-01/02 challenges are based on. These challenges are fatally flawed in their current form, and should be discontinued. (Your solution in the first sentence changes it to domain control = domain control. Obviously correct.)
tonyarkles 9y ago
> it would seem fundamentally incorrect to make a request with a Host header or SNI hostname set to anything other than the hostname requested for issuance. Pondering why they'd do it the way it is right now, my best guess would be that it allows you to get a LetsEncrypt certificate while still having a valid certificate from another CA. For example, example.com currently has a Symantec certificate that is soon going to be invalidated, and I want to swap over to an LE cert with 0 downtime.
- BillinghamJ 9y ago
  You could still do that! Just accept LE’s requests with the existing Symantec cert and respond as needed. Or use the DNS challenge. Or use something like ALPN instead.
  - tonyarkles 9y ago
    How would LE do SNI validation using the Symantec cert though? I can't add anything to it without invalidating it. My understanding is that that's why they do the .acme.invalid SNI responses; so that the server can respond with a self-signed cert that's not going to interfere with anything else. DNS and ALPN are both great solution, but here's why neither of them work all that well in my specific circumstances, but TLS-SNI works awesome. - I can provision VMs at will, but I don't personally have access to DNS. To get a DNS record added, I need to submit a change request. I haven't done the DNS challenge, and I'm not sure if a new TXT record gets generated when you renew a cert or if it's static after initial creation, but either way, DNS very much adds a painful manual step to cert provisioning here. - ALPN would be great, if it were baked into the default Centos Apache :). That, I suspect, would be a long time coming... Right now I generally just use HTTP validation, because I've worked out a pattern that generally works pretty good. I set up an Apache rule that auto-redirects all traffic from http to https anyway; I just add a condition to it so that /.well-known/ gets served off the filesystem before redirecting everything else.

rgbrenner 9y ago

How does TLS-SNI-02 fix this issue? ... Edit: I've removed most of my post here... This was already acknowledged in the article. 02 is vulnerable also.

mdhardeman 9y ago
I think the only fix is to create TLS-SNI-03 in which the only dnsName component in the self-signed certificate is a well known child of the domain label to be validated. Validating www.abc.com, SNI and dnsName is well-known-acme-pki.www.abc.com and the certificate should have some other parameter stuffed with a challenge response that is defined to be a challenge response calculated over the inputs of a random token provided to the requestor by the CA and the account key of the requestor.

mappu 9y ago

This is not specifically a vulnerability in TLS-SNI-01 - more like a major misconfiguration by the shared hosting provider, to share ACME secrets. I guess TLS-SNI-01 does share part of the blame for making this possible in the first place. It would be great if TLS-SNI-01 could be salvaged, for all the binaries out there - but i don't think a blacklist is a great long-term solution. How about a strict one IP = one domain lockdown until you can prove that you don't share challenges? i.e. negative responses. EDIT: See reply

rgbrenner 9y ago
I disagree. It's a fatal flaw in TLS-SNI-01/02. Here's why: All of the information required to solve the challenge is in the attackers control. SAN-A and SAN-B (in v02) are known by the attacker. And no action or information from the target domain is required to solve the challenge. Every shared hosting provider and CDN that lets you pick a domain and upload a custom certificate can be used to solve the challenge. Your solution of restricting it to ip = one domain just reduces the vulnerable sites to sites that don't use LE and are on a shared provider. (Also note, if the provider does not implement LE.. then they are vulnerable to this attack against all of the customers.. with your solution: 1 customer for each IP owned by the service) As long as SNI is part of the TLS standard, it is incorrect to assume IP = domain ownership.
- mappu 9y ago
  You're right. Thanks for clarifying.

tialaramex 9y ago

My _guess_ at this point is that there's a bunch of scary corner cases with the behaviour that makes this possible so that if your bulk host or CDN does this you need to be _very_ careful even if Let's Encrypt abolishes tls-sni family proofs of control. If I'm right, fixing this is pretty urgent for anyone with untrusted users sharing an IP address. It looks very much as if affected hosts/CDNs are letting any customer impersonate any other customer with the only caveats being that you've got to get in first and you need the victim customer to make the config change that causes this to be useful. Suppose I am victim.example, I have a contract with Funky Jim's Hosting, and I have configured *.clowns.victim.example to go to a server at Funky Jim's but I've only every bothered setting up www.clowns.victim.example, images.clown.victim.example and js.clowns.victim.example in Funky Jim's config system. I use HTTPS, so the valuable Clown Login tokens available on www.clowns.victim.example are protected - aren't they? Some loser, let's call her Sad Carol buys a basic account from Funky Jim, and tells Funky Jim they want to add a new site, busted.clowns.victim.example. Funky Jim OKs it, because that's "harmless" right? No need for Carol to prove to Jim that she owns it, if not why would it matter? But wait, actually busted.clowns.victim.example matches my wildcard DNS entry, it goes to the right server and serves up Carol's site - now Sad Carol can get all my bloody Clown Login cookies sent to "her" web site just by injecting "her" site into an ad inventory or whatever.

mdhardeman 9y ago
An excellent example of a plausible attack scenario. Having said that, I think what we're all dying to know is what was it, initially, that made Carol so sad? And, indeed, was it that same sadness which stagnated, fermented, and evolved into the obvious hatred and antisocial behaviors that she exhibits here?

nodesocket 9y ago

> At this time, we believe that the issue can be addressed by having certain services providers implement stronger controls for domains hosted on their infrastructure. Is DigitalOcean one of these service providers that need to implement stronger controls?