5 ms·

Josh from Let's Encrypt here. I'm not able to give many more details yet, but here's what I can add now: 1) This isn't a relatively simple issue like a bug in

by jaas 9y ago

Josh from Let's Encrypt here. I'm not able to give many more details yet, but here's what I can add now: 1) This isn't a relatively simple issue like a bug in our CA code would be. It's an interaction between the protocol and provider services. 2) Disabling TLS-SNI is a complete mitigation for us, meaning it's no longer possible to get an illegitimate certificate from Let's Encrypt by exploiting this issue. 3) We have not yet reached a conclusion as to whether or not the TLS-SNI challenge will need to remain disabled permanently. 4) At this point we have no reason to believe that the vulnerability has been exploited by anyone other than the researcher who figured it out and reported it to us. Our focus now is on sharing information with relevant parties and looking for less drastic mitigations that might allow us to restore the TLS-SNI challenge option to people who rely on it. We will, of course, share more information as soon as we can. That might be as soon as the next few hours, things are moving quickly.

niftich 9y ago
Thanks for the update and good work. Are you able to share and/or clarify whether the TLS-SNI-02 challenge is affected as well, in addition to TLS-SNI-01 which is currently Production on LE?
- jaas 9y ago
  Haven't put much thought into -02 since nobody actually runs it in production but I suspect the same issue applies.
culerawo 9y ago
Thanks for sharing this! Please share it on your blog as well https://letsencrypt.org/blog/ https://letsencrypt.org/blog/ I subscribed the RSS to keep me up-to-date
terom 9y ago
Do we get points for speculation based on these hints? My guess would be that some major public CDN (Cloudflare etc) will let the attacker deploy their TLS-SNI challenge certs, and thus validate for other victim domains using the same CDN service. EDIT: Main reasoning being that I can't think why it wouldn't work - apart from the TLS-SNI challenge certs somehow being considered invalid by the CDN provider and refusing to deploy them, but I find it hard to trust that happening with 100% certainty.
- abofh 9y ago
  As long as we're tossing hats into the ring... I don't think so - _maybe_ a MITM that would affect all TLS-SNI, but I don't think the model you propose exists (I'm a bit dated on that tho), and if it did, I think it would be 'wider' than the CDN's. "It's an interaction between the protocol and provider services" -- I will wager it's that some providers don't validate the CA or RootCA authentication method/mechinism, and so would accept a SHA1 or MD5 where a SHA256 or ROT13 would be preferred. (Think JWK/JWS 'null' issues) I assume standard rules apply - one internet point to the winner, in the event of a draw, fisticuffs at dawn facing opposite coasts until we're both bored with it?
  - foota 9y ago
    "and so would accept a SHA1 or MD5 where a SHA256 or ROT13 would be preferred" I don't think ROT13 means what you think it means.
- PuffinBlue 9y ago
  I suppose it's possible in theory but I'm not sure if Cloudflare would perform other checks. It's certainly the case with the Business and above accounts you can upload custom certificates but I don't have one of those accounts to check if further validation is made regarding domain control. If they didn't it would seem on the face of it that both the conditions in the article* are met. * >Many users are hosted on the same IP address >Users have the ability to upload certificates for arbitrary names without proving domain control.
- jgrahamc 9y ago
  I haven't seen anything on our internal security mailing lists about this. If it does somehow involve Cloudflare I'd be happy to receive a report directly via HackerOne (https://hackerone.com/cloudflare https://hackerone.com/cloudflare). Happy to assist.
  - terom 9y ago
    Sorry, I just chose cloudflare as a random example when speculating, I don't have any information about what specific providers this would affect, and I'm not implying that cloudflare would be affected. Seems like my guess was right, though!
mdhardeman 9y ago
I just read the details of the tls-sni-01 challenge again. I can certainly see a problem. With respect to the final validation stage, where the validator is communicating with the TLS endpoint, all the information needed to satisfy the validator is presented IN the SNI request data. It looks like anyone could request a tls-sni-01 validation with any account key, for a host that they new would resolve to a TLS server infrastructure with a certain behavior and in turn be able to get the certificate issued. The qualifying behavior would be a TLS server infrastructure which has some sort of opportunistic generation of a self-sign certificate for an unrecognized SNI name, such that the just-in-time minted certificate would have a single SAN dnsName being that exact value requested in the SNI. Unlike the other challenges where the final validating query from the VA does NOT contain the "answer" needed, it would appear that the in the final pass of the TLS-SNI-01 protocol exchange, the final "answer" is indeed in the request. You know, I'm trying to recall the details, but I'm pretty sure I came across a consumer router platform once that, until and unless administratively configured with a certificate, would keep opportunistically recreating certificates to match the SNI name. This was to make the only validation failure be "signed by an untrusted..." instead of "name mismatch", etc, etc.... I wonder if a critical mass of things like that has arisen?
- pfg 9y ago
  If tls-sni-02 is affected too (as Josh indicated it probably is below), I'd suspect something like a major CDN or hosting provider allowing deployment of arbitrary, attacker-controlled certificates on arbitrary domains under "acme.invalid". That would be the only thing I can think of that would bypass the "anti-parrot" mitigation introduced in -02. Otherwise, it's probably what you're describing.
  - mdhardeman 9y ago
    I can totally believe that may exist too. After all, for the CDN / Hosting company, what's the real risk? It wouldn't shock me if lots of infrastructure lets you board a new and novel host label that hasn't already been boarded and starts letting you configure things like a custom TLS certificate to be associated with that label. As long as they watch out for uniqueness, what's the risk to the CDN? That a config context gets created for a DNS label that isn't yet pointed there? It's not an obvious risk from the CDN's perspective. Even for an invalid domain. "Oh, sad, I just wasted mere kilobytes of storage on a configuration for a domain that you're never going to actually be able to get into the DNS and point to me?" That sounds kind of low cost, from the CDN's perspective.
vog 9y ago
Curious question: Why do you write those details here on HN and not on the Letsencrypt Blog or letsencrypt.status.io? Are HN readers somehow more entitled to these information that those who "merely" subscribed to your blog's RSS feed?
- nindalf 9y ago
  I agree that a blog post would be more helpful, but the barrier to creating one is higher. This sort of update wouldn't fly there because its an off-the-cuff status report that will be buried in a day or two. The Let's Encrypt blog will be available and readable for all eternity (give or take a few years), so it requires an update that's more in-depth and vetted by all the people involved in solving this issue. I'd imagine that once they fix this issue the first thing they'll do is to post a post-mortem on their blog.
  - vog 9y ago
    I see. So this is more to be interpreted as a rushed preliminary report, forced upon them by appearing at the HN front page, to prevent uninformed rumor to appear and to spread. Or, did I miss something? EDIT: Apparently the latter didn't work, as the very first response to their comment starts with "Do we get points for speculation based on these hints? ..."
    - tialaramex 9y ago
      It prevented _uninformed_ rumours :D Almost all the speculation was about the exact scenario that Let's Encrypt have subsequently confirmed - you can trick some bulk hosting or CDN providers into letting you (a customer) answer a challenge for one of their other customers' systems.
regecks 9y ago
New Update, explaining the issue with shared infrastructure and crediting the discovery to Frans Rosén of Detectify: https://community.letsencrypt.org/t/2018-01-09-issue-with-tls-sni-01-and-shared-hosting-infrastructure/49996 https://community.letsencrypt.org/t/2018-01-09-issue-with-tl...

by jaas 9y ago

niftich 9y ago

Thanks for the update and good work. Are you able to share and/or clarify whether the TLS-SNI-02 challenge is affected as well, in addition to TLS-SNI-01 which is currently Production on LE?

jaas 9y ago
Haven't put much thought into -02 since nobody actually runs it in production but I suspect the same issue applies.

culerawo 9y ago

Thanks for sharing this! Please share it on your blog as well https://letsencrypt.org/blog/ https://letsencrypt.org/blog/ I subscribed the RSS to keep me up-to-date

terom 9y ago

Do we get points for speculation based on these hints? My guess would be that some major public CDN (Cloudflare etc) will let the attacker deploy their TLS-SNI challenge certs, and thus validate for other victim domains using the same CDN service. EDIT: Main reasoning being that I can't think why it wouldn't work - apart from the TLS-SNI challenge certs somehow being considered invalid by the CDN provider and refusing to deploy them, but I find it hard to trust that happening with 100% certainty.

abofh 9y ago
As long as we're tossing hats into the ring... I don't think so - _maybe_ a MITM that would affect all TLS-SNI, but I don't think the model you propose exists (I'm a bit dated on that tho), and if it did, I think it would be 'wider' than the CDN's. "It's an interaction between the protocol and provider services" -- I will wager it's that some providers don't validate the CA or RootCA authentication method/mechinism, and so would accept a SHA1 or MD5 where a SHA256 or ROT13 would be preferred. (Think JWK/JWS 'null' issues) I assume standard rules apply - one internet point to the winner, in the event of a draw, fisticuffs at dawn facing opposite coasts until we're both bored with it?
- foota 9y ago
  "and so would accept a SHA1 or MD5 where a SHA256 or ROT13 would be preferred" I don't think ROT13 means what you think it means.
PuffinBlue 9y ago
I suppose it's possible in theory but I'm not sure if Cloudflare would perform other checks. It's certainly the case with the Business and above accounts you can upload custom certificates but I don't have one of those accounts to check if further validation is made regarding domain control. If they didn't it would seem on the face of it that both the conditions in the article* are met. * >Many users are hosted on the same IP address >Users have the ability to upload certificates for arbitrary names without proving domain control.
jgrahamc 9y ago
I haven't seen anything on our internal security mailing lists about this. If it does somehow involve Cloudflare I'd be happy to receive a report directly via HackerOne (https://hackerone.com/cloudflare https://hackerone.com/cloudflare). Happy to assist.
- terom 9y ago
  Sorry, I just chose cloudflare as a random example when speculating, I don't have any information about what specific providers this would affect, and I'm not implying that cloudflare would be affected. Seems like my guess was right, though!

mdhardeman 9y ago

I just read the details of the tls-sni-01 challenge again. I can certainly see a problem. With respect to the final validation stage, where the validator is communicating with the TLS endpoint, all the information needed to satisfy the validator is presented IN the SNI request data. It looks like anyone could request a tls-sni-01 validation with any account key, for a host that they new would resolve to a TLS server infrastructure with a certain behavior and in turn be able to get the certificate issued. The qualifying behavior would be a TLS server infrastructure which has some sort of opportunistic generation of a self-sign certificate for an unrecognized SNI name, such that the just-in-time minted certificate would have a single SAN dnsName being that exact value requested in the SNI. Unlike the other challenges where the final validating query from the VA does NOT contain the "answer" needed, it would appear that the in the final pass of the TLS-SNI-01 protocol exchange, the final "answer" is indeed in the request. You know, I'm trying to recall the details, but I'm pretty sure I came across a consumer router platform once that, until and unless administratively configured with a certificate, would keep opportunistically recreating certificates to match the SNI name. This was to make the only validation failure be "signed by an untrusted..." instead of "name mismatch", etc, etc.... I wonder if a critical mass of things like that has arisen?

pfg 9y ago
If tls-sni-02 is affected too (as Josh indicated it probably is below), I'd suspect something like a major CDN or hosting provider allowing deployment of arbitrary, attacker-controlled certificates on arbitrary domains under "acme.invalid". That would be the only thing I can think of that would bypass the "anti-parrot" mitigation introduced in -02. Otherwise, it's probably what you're describing.
- mdhardeman 9y ago
  I can totally believe that may exist too. After all, for the CDN / Hosting company, what's the real risk? It wouldn't shock me if lots of infrastructure lets you board a new and novel host label that hasn't already been boarded and starts letting you configure things like a custom TLS certificate to be associated with that label. As long as they watch out for uniqueness, what's the risk to the CDN? That a config context gets created for a DNS label that isn't yet pointed there? It's not an obvious risk from the CDN's perspective. Even for an invalid domain. "Oh, sad, I just wasted mere kilobytes of storage on a configuration for a domain that you're never going to actually be able to get into the DNS and point to me?" That sounds kind of low cost, from the CDN's perspective.

vog 9y ago

Curious question: Why do you write those details here on HN and not on the Letsencrypt Blog or letsencrypt.status.io? Are HN readers somehow more entitled to these information that those who "merely" subscribed to your blog's RSS feed?

nindalf 9y ago
I agree that a blog post would be more helpful, but the barrier to creating one is higher. This sort of update wouldn't fly there because its an off-the-cuff status report that will be buried in a day or two. The Let's Encrypt blog will be available and readable for all eternity (give or take a few years), so it requires an update that's more in-depth and vetted by all the people involved in solving this issue. I'd imagine that once they fix this issue the first thing they'll do is to post a post-mortem on their blog.
- vog 9y ago
  I see. So this is more to be interpreted as a rushed preliminary report, forced upon them by appearing at the HN front page, to prevent uninformed rumor to appear and to spread. Or, did I miss something? EDIT: Apparently the latter didn't work, as the very first response to their comment starts with "Do we get points for speculation based on these hints? ..."
  - tialaramex 9y ago
    It prevented _uninformed_ rumours :D Almost all the speculation was about the exact scenario that Let's Encrypt have subsequently confirmed - you can trick some bulk hosting or CDN providers into letting you (a customer) answer a challenge for one of their other customers' systems.

regecks 9y ago

New Update, explaining the issue with shared infrastructure and crediting the discovery to Frans Rosén of Detectify: https://community.letsencrypt.org/t/2018-01-09-issue-with-tls-sni-01-and-shared-hosting-infrastructure/49996 https://community.letsencrypt.org/t/2018-01-09-issue-with-tl...