6 ms·

Over the next 48 hours we will be building a list of vulnerable providers and their associated IP addresses. Our tentative plan, once the list is completed, is

Over the next 48 hours we will be building a list of vulnerable providers and their associated IP addresses. Our tentative plan, once the list is completed, is to re-enable the TLS-SNI-01 challenge type with vulnerable providers blocked from using it. This is a terrible solution. You're assuming you can scour the internet and find every shared hosting provider affected, and add them to a list. And then you're also going to keep that list updated. That's crazy.. an impossible task. So after this is done, Let's Encrypt will continue issuing certificates improperly on any shared hosting providers that didn't get added to this list. Attackers just need to find a provider and check a list now. I'm actually kind of appalled that this is what you've come up with.

deleted 9y ago
[deleted]
mdhardeman 9y ago
I concur in full with your commentary here. LetsEncrypt, through thoughtful planning and action -- even when it was inconvenient for them -- has established themselves as a shining beacon of best practice. For better or worse, when you become the best practice example, you are held by the world -- and especially by your peers -- to a higher standard. I can not imagine how they would see a return to production status of TLS-SNI-01 as anything other than painting a target on themselves for their various detractors to take aim at. It is no secret that numerous other parties in the CA space would love to see them slip and fall.
DanielDent 9y ago
I think this issue can be described as a form of confused deputy problem. Is not the real solution to have the confused deputies stop acting in a confused manner? If this issue were confined to a handful of small hosting companies almost no one uses, then I don't think this would be described as a problem with Let's Encrypt - it would be described as a problem with those hosting companies. The only party that can truly fix the issue of the incorrect user being granted access (directly or indirectly) to a domain pointed at shared infrastructure is the operator of that infrastructure.
- deleted 9y ago
  [deleted]
- e_d_e_v 9y ago
  I came here to say this. What's more, the spec was agreed upon, in relatively public forums, with a voice from the community. Crappy shared hosting providers are going to mostly ignore their customers and perpetuate insecure scenarios while they continue to bill exorbitant rates that exploit the customers' ignorance or inertia. That has been the case for some time, and will continue to be the case, this is just another symptom.
  - rgbrenner 9y ago
    What's more, the spec was agreed upon, in relatively public forums, with a voice from the community. It was agreed upon and no one caught this issue. Now we know the issue. There's nothing wrong with using a protocol you think is correct. There is something wrong with using a protocol you know is incorrect, but continue to use it anyway. The entire internet should not be required from now to forever to workaround LE's mistake. LE should fix their protocol. And worse, this protocol isn't even needed for LE. They could remove it, and everyone could use one of the two others that are secure, and LE would be just fine, and everyone -- even those crappy shared hosting providers -- would be perfectly secure. LE created this issue all by itself, and is capable of fixing it all by itself. LE should do that.
    - DanielDent 9y ago
      LE "created" this issue in the sense that they were the first to formalize an implementable specification for automated verification of authorized domain name use. In comparison to the prior relatively unspecified approach to verification, it's still an improvement. "The last person who touched needs to take ownership over anything anyone can blame on the change" is the management style which leads to enterprise IT being unable to get anything done. Because at that point, the safest thing is to never change anything - regardless of how bad things currently are. Sometimes the world changes, and other parts of the technology ecosystem need to adapt. [Editing to add since I can't reply to you]: The fundamental "flaw" here is that it's possible for people to get self-signed certificates served for domains where they haven't validated ownership of the domain. Hosting companies can't be simply adjusting their routing tables for anyone who asks. If you are pointing a domain name at an IP address which will accept routes from any untrusted party, that's simply not a secure situation. A signed certificate might be good evidence of some authority for a domain, but a self-signed certificate used in a challenge process most assuredly is not.
e_d_e_v 9y ago
> You're assuming you can scour the internet and find every shared hosting provider affected, and add them to a list. And then you're also going to keep that list updated. That's crazy.. an impossible task. I think you underestimate the capabilities of modern infosec tooling. Essentially the whole of the internet can be scanned in some ways in durations measured in hours. What is more, some systems are being constantly updated (such a certificate transparency), and there are relatively easy ways to identify bad actors via whitelists and behavioral monitoring. All that being said, if you have a legitimately better idea, voice it in a meaningful way, and I am sure they will at least listen. That was the part _in_their_post_ about "taking community feedback" you must have missed.
- rgbrenner 9y ago
  The underlying assumption of TLS-SNI-01/02 is that IP ownership = ownership of all domains using that IP. But as long as SNI is part of the TLS standard, that is an incorrect assumption. So the affected hosts that need to be in this list is every shared hosting IP -- not just those that use LE, but all of them. They can then remove IPs once they verify LE-specific workarounds have been applied. But they need to scan IPv6 as well.. since, although SNI is less necessary with IPv6, it can still be used with IPv6. So they need to scan the entire IPv4 and IPv6 address space (regularly.. to keep the list updated) looking for shared IPs. And we'll assume their scanner won't be blocked or interfered with in any way, so they'll actually be able to determine which IPs are being shared. As far as a solution, I think I've been very clear in my other posts. The underlying assumption in this protocols is incorrect, and that makes them fatally flawed. They should be discontinued entirely (at least until we've all moved to IPv6 and SNI has been removed from the TLS standard).
  - dec0dedab0de 9y ago
    Since the issue is only domain based, Could they start with a list of domain names instead of every IP?
    - rgbrenner 9y ago
      They would need to resolve all of the domains, and compile a list of IPs.. but one problem that jumps out to me is geodns/round robin policies/etc. If LE makes a dns request, they can't be guaranteed that they're seeing all of the IPs for a domain. For example.. if a number of domains are at a CDN (that does not use anycast)... they may all resolve to a single IP (to LE, from the location they're requesting from)... but really that CDN may have hundreds of IPs that are all valid for those domains. LE would then add that single IP to the list as a shared IP, but LE verification requests sent to those other IPs would still be vulnerable.
ploxiln 9y ago
The only domains vulnerable to attackers are those pointed at the shared hosting provider with this issue. So you can protect your own domain by hosting it with a provider you know does not have this issue. And if a shared provider has this issue, they might have others which compromise the isolation of accounts/domains. And major websites which host themselves are not at any risk.
- mdhardeman 9y ago
  I agree with your position entirely, save for the "and if a shared provider has this issue, they might..." No one ever told the hosted service providers that they should explicitly guard for TLS SNI names unrelated to any name of the customers they host. That certainly doesn't follow from any obvious logic. I don't believe that a service provider who has this susceptibility is necessarily any more likely to surface other threats.

deleted 9y ago

[deleted]

mdhardeman 9y ago

I concur in full with your commentary here. LetsEncrypt, through thoughtful planning and action -- even when it was inconvenient for them -- has established themselves as a shining beacon of best practice. For better or worse, when you become the best practice example, you are held by the world -- and especially by your peers -- to a higher standard. I can not imagine how they would see a return to production status of TLS-SNI-01 as anything other than painting a target on themselves for their various detractors to take aim at. It is no secret that numerous other parties in the CA space would love to see them slip and fall.

DanielDent 9y ago

I think this issue can be described as a form of confused deputy problem. Is not the real solution to have the confused deputies stop acting in a confused manner? If this issue were confined to a handful of small hosting companies almost no one uses, then I don't think this would be described as a problem with Let's Encrypt - it would be described as a problem with those hosting companies. The only party that can truly fix the issue of the incorrect user being granted access (directly or indirectly) to a domain pointed at shared infrastructure is the operator of that infrastructure.

deleted 9y ago
[deleted]
e_d_e_v 9y ago
I came here to say this. What's more, the spec was agreed upon, in relatively public forums, with a voice from the community. Crappy shared hosting providers are going to mostly ignore their customers and perpetuate insecure scenarios while they continue to bill exorbitant rates that exploit the customers' ignorance or inertia. That has been the case for some time, and will continue to be the case, this is just another symptom.
- rgbrenner 9y ago
  What's more, the spec was agreed upon, in relatively public forums, with a voice from the community. It was agreed upon and no one caught this issue. Now we know the issue. There's nothing wrong with using a protocol you think is correct. There is something wrong with using a protocol you know is incorrect, but continue to use it anyway. The entire internet should not be required from now to forever to workaround LE's mistake. LE should fix their protocol. And worse, this protocol isn't even needed for LE. They could remove it, and everyone could use one of the two others that are secure, and LE would be just fine, and everyone -- even those crappy shared hosting providers -- would be perfectly secure. LE created this issue all by itself, and is capable of fixing it all by itself. LE should do that.
  - DanielDent 9y ago
    LE "created" this issue in the sense that they were the first to formalize an implementable specification for automated verification of authorized domain name use. In comparison to the prior relatively unspecified approach to verification, it's still an improvement. "The last person who touched needs to take ownership over anything anyone can blame on the change" is the management style which leads to enterprise IT being unable to get anything done. Because at that point, the safest thing is to never change anything - regardless of how bad things currently are. Sometimes the world changes, and other parts of the technology ecosystem need to adapt. [Editing to add since I can't reply to you]: The fundamental "flaw" here is that it's possible for people to get self-signed certificates served for domains where they haven't validated ownership of the domain. Hosting companies can't be simply adjusting their routing tables for anyone who asks. If you are pointing a domain name at an IP address which will accept routes from any untrusted party, that's simply not a secure situation. A signed certificate might be good evidence of some authority for a domain, but a self-signed certificate used in a challenge process most assuredly is not.

e_d_e_v 9y ago

> You're assuming you can scour the internet and find every shared hosting provider affected, and add them to a list. And then you're also going to keep that list updated. That's crazy.. an impossible task. I think you underestimate the capabilities of modern infosec tooling. Essentially the whole of the internet can be scanned in some ways in durations measured in hours. What is more, some systems are being constantly updated (such a certificate transparency), and there are relatively easy ways to identify bad actors via whitelists and behavioral monitoring. All that being said, if you have a legitimately better idea, voice it in a meaningful way, and I am sure they will at least listen. That was the part _in_their_post_ about "taking community feedback" you must have missed.

rgbrenner 9y ago
The underlying assumption of TLS-SNI-01/02 is that IP ownership = ownership of all domains using that IP. But as long as SNI is part of the TLS standard, that is an incorrect assumption. So the affected hosts that need to be in this list is every shared hosting IP -- not just those that use LE, but all of them. They can then remove IPs once they verify LE-specific workarounds have been applied. But they need to scan IPv6 as well.. since, although SNI is less necessary with IPv6, it can still be used with IPv6. So they need to scan the entire IPv4 and IPv6 address space (regularly.. to keep the list updated) looking for shared IPs. And we'll assume their scanner won't be blocked or interfered with in any way, so they'll actually be able to determine which IPs are being shared. As far as a solution, I think I've been very clear in my other posts. The underlying assumption in this protocols is incorrect, and that makes them fatally flawed. They should be discontinued entirely (at least until we've all moved to IPv6 and SNI has been removed from the TLS standard).
- dec0dedab0de 9y ago
  Since the issue is only domain based, Could they start with a list of domain names instead of every IP?
  - rgbrenner 9y ago
    They would need to resolve all of the domains, and compile a list of IPs.. but one problem that jumps out to me is geodns/round robin policies/etc. If LE makes a dns request, they can't be guaranteed that they're seeing all of the IPs for a domain. For example.. if a number of domains are at a CDN (that does not use anycast)... they may all resolve to a single IP (to LE, from the location they're requesting from)... but really that CDN may have hundreds of IPs that are all valid for those domains. LE would then add that single IP to the list as a shared IP, but LE verification requests sent to those other IPs would still be vulnerable.

ploxiln 9y ago

The only domains vulnerable to attackers are those pointed at the shared hosting provider with this issue. So you can protect your own domain by hosting it with a provider you know does not have this issue. And if a shared provider has this issue, they might have others which compromise the isolation of accounts/domains. And major websites which host themselves are not at any risk.

mdhardeman 9y ago
I agree with your position entirely, save for the "and if a shared provider has this issue, they might..." No one ever told the hosted service providers that they should explicitly guard for TLS SNI names unrelated to any name of the customers they host. That certainly doesn't follow from any obvious logic. I don't believe that a service provider who has this susceptibility is necessarily any more likely to surface other threats.