7 ms·

The link is very technical, resulting in some confusion as to why this is such a big problem. The comments on HN reflect that. Here’s my understanding: This is

The link is very technical, resulting in some confusion as to why this is such a big problem. The comments on HN reflect that. Here’s my understanding: This isn’t a problem because a sub-CA can revoke any certificate from any other sub-CA of the same CA. That would be bad, but, at worst, it’s denial-of-service. Rather, this is a problem because any sub-CA can effectively reverse the revocation of any other sub-CA, or the CA itself. That’s immensely problematic. Suddenly, the CA has no reliable way fully revoke certificates. Revocation is already somewhat broken as it is, but this gives a lot of entities the ability to deliberately interfere with revocation in ways that they shouldn’t be able to. The author goes on to explain that revocation of the affected certificates is insufficient, because they could be used to effectively reverse their own revocation at any point in the future. Instead, it must be proven that all copies of the keys have been destroyed. That’s quite an undertaking. What the author fails to mention is that revocation is already pretty broken. Most major browsers have their own built-in CRL replacements that contain the most important revocations they need to know about. Some browsers, like Firefox, may make additional efforts to ensure that any given certificate hasn’t been revoked; others, like Chrome, don’t. If you’ve ever visited a site that gives you a certificate error in Firefox but not Chrome, that’s likely why. In the case of browsers, it should be possible for each browser to forcefully revoke affected certificates, but revoking a sub-CA certificate is quite disruptive, so I’d be surprised if that happens within 7 days. The catch is that this technique is really only effective in modern, up-to-date browsers. In any case, the title is misleading. I don’t see where the author guarantees that this will happen within 7 days. The author claims it should happen within 7 days, but considering that the damage is already done and cannot be fully reversed by revocation, I find it hard to believe enforcing that deadline makes sense here.

zenexer 6y ago
Flagging for a title change, as the revocation deadline was proposed by a single individual on a mailing list, and there’s no evidence from further discussion in that thread that a consensus has been reached. I would not expect all these certificates to be revoked within 7 days.
- mbrinkers 6y ago
  I changed the title. Although I heard from someone involved that the intermediates really should be revoked in 7 days. Let's wait and see.
  - zenexer 6y ago
    They definitely should be—that’s what the author is claiming is mandated, and it would make sense. However, I’m a bit skeptical about browsers being able to enforce that timeline here. Also, given that the underlying cause appears to be ignorance, it would be prudent to take things slow and ensure that this doesn’t happen again. As I said before, the damage is already done—revoking appears to be insufficient here. If this does actually happen within 7 days, though, I will be thoroughly impressed.
    - floatingatoll 6y ago
      It could be considered a tacit warning that browsers may choose to mistrust the impacted subCAs in the near future. I don’t know the specifics, but I assume they can revoke for non-compliance using in-browser mechanisms without depending on the revocation process. EDIT: Mozilla’s reply: https://news.ycombinator.com/item?id=23748561 https://news.ycombinator.com/item?id=23748561
    - tptacek 6y ago
      You don't have to trust Sleevi (though: you always should); you can just read the BRs. The revocation requirement is in this case black-letter SHALL. https://cabforum.org/wp-content/uploads/CA-Browser-Forum-BR-1.7.0-1.pdf https://cabforum.org/wp-content/uploads/CA-Browser-Forum-BR-...
- EE84M3i 6y ago
  That "single individual on a mailing list" is Ryan Sleevi, who is basically the voice of Google in all matters to do with TLS PKI, and as I understand it is not working with a "opinions my own" cop-out in this context. It's not some rando
neallindsay 6y ago
"...revocation of the affected certificates is insufficient... Instead, it must be proven that all copies of the keys have been destroyed." Isn't the main reason you would want to revoke keys because they were disclosed, making it impossible to destroy all copies?
- jakear 6y ago
  Further, if I can destroy all keys do I even need to revoke the cert? (Honest question, I’m actually not sure)
  - floatingatoll 6y ago
    Revocation is required by policy, so the question is technically moot. It’s generally good practice to generate and publish a revocation prior to destroying a private key, though. To provide an analogy in the context of PGP keys, if an attacker somehow finds a backup of your revoked and destroyed private key someday, they will have trouble using it because your revocation will be public and on record.
- floatingatoll 6y ago
  Normally, yes. This is not a normal circumstance though. In this scenario, the misissued intermediates effectively have sudo access to cancel a revocation issued by the parent CA. This is equivalent to being told that a non-root user could cancel a userdel command run by root. For policy compliance, the intermediates have to revoke their certificates — but since the intermediates can immediately un-revoke themselves, proof of key destruction is necessary as well to ensure that they cannot.
gruez 6y ago
>The author goes on to explain that revocation of the affected certificates is insufficient, because they could be used to effectively reverse their own revocation at any point in the future. Instead, it must be proven that all copies of the keys have been destroyed. That’s quite an undertaking. How would this be verified? Presumably the keys are stored on HSMs, but you can I'm not sure how you can prove that you didn't make a backup of the key.
- jchrisa 6y ago
  The point is that you can't verify it. Sometimes HN humor is too dry for its own good.
- michaelt 6y ago
  In principle, you would hope a CA would be able to precisely account for the number of backup copies of their private key. In practice, of course, that doesn't mean every one of them will have done. There's 293 of them, after all.
- floatingatoll 6y ago
  It’s impossible to prevent a truly dedicated malicious actor from doing so, but enforcement through both policy and independent auditors — and quality of response to security incidents — provide several layers of defenses against this scenario. (As with all things, a perfect defense is ultimately impossible, but they put in a lot of effort to get a lot of nines of certainty.)
- TrueDuality 6y ago
  It is largely impossible to fully prove. CAs are supposed to keep detailed records of any issuing keys and what was called for was specifically "witnessed Key Destruction Reports" which involves third party independent confirmation of destruction of documented keys. In the event that a key with a Key Destruction Report shows up again, the responsible party for that key will have shown unacceptable negligence and will potentially be subject to the exclusion of their keys as a valid certificate signer. A lot of these companies core businesses rely on remaining in a position to sign certificates so it is in their best interest to protect that privilege by following the documentation requirements, and properly destroy their keys. It's effectively a pretty good stick.
xZuki 6y ago
This is all confusing to me but I've been having certificate issues today and it seems like this could be related. It's a weird coincidence if not! Basically on Chrome one of my sites is throwing: "NETT::ERR_CERTIFICATE_TRANSPARENCY_REQUIRED" for most users but not all, even though they're all on Chrome. It seems to work fine in other browsers. https://transparencyreport.google.com/https/certificates https://transparencyreport.google.com/https/certificates When I check my domain here it seems like I have got the transparency certificate so I shouldn't be getting this error. Is this related to what you're talking about? I would really appreciate any help. I'm using https://letsencrypt.org/ https://letsencrypt.org/ for the cert.
- geofft 6y ago
  Does your cert have an SCT? It would be strange for a Let's Encrypt cert to be missing it but certainly possible. Try running (replace both example.coms with your domain name) openssl s_client -connect example.com:443 -servername example.com </dev/null | openssl x509 -noout -text which should print an SCT extension at the end - my version displays it by numeric identifier "1.3.6.1.4.1.11129.2.4.2" but maybe newer versions display it by name. Alternatively, I think you might able to go to https://www.ssllabs.com/ssltest/ https://www.ssllabs.com/ssltest/ and see if your cert has "Certificate Transparency: Yes", but I'm not sure exactly what that means. Anyway, I don't think this is related, the question at hand is about OCSP, which is a different mechanism from Certificate Transparency. (Arguably Certificate Transparency is a replacement for revocation in general being flawed in practice for many reasons, but they're different mechanisms.)
- floatingatoll 6y ago
  That is probably not related. If you run SSLLabs against your host name, does it say “Certificate Transparency: Yes” or No? https://www.ssllabs.com/ssltest/analyze.html?d=your-hostname https://www.ssllabs.com/ssltest/analyze.html?d=your-hostname
  - xZuki 6y ago
    Thanks. It says "yes".
btown 6y ago
What are the state-actor-level attack implications of this? Before this was revealed, a party that compromised (or was able to be issued) a certificate for a website could be reasonably likely to be detected and have that certificate revoked if they used it for large-scale MITM or redirection. But now, if the actor were to also compromise any one of these sub-CAs before the key was deleted, could they permanently be in a position to be able to unilaterally reverse any such revocations, effectively giving them carte blanche to begin a campaign of compromising websites in earnest with the knowledge that their attacks would now be "sticky?" What would the recourse be here if one of those keys were to be compromised, or even if there was reason to believe one might have been? Would the entire CA-level trust chain need to be distrusted, requiring re-issuance of all certificates on that chain?
- geofft 6y ago
  Yes and almost-yes - you don't need to destroy the root CA itself, but you need to physically destroy the keys for any affected subordinate CAs. (Revoking them isn't sufficient, because they can just un-revoke themselves.) From the post: "... the degree of this issue likely also justifies requiring witnessed Key Destruction Reports, in order to preserve the integrity of the issuer of these certificates (which may include the CA's root)." The "good" news is that most people haven't really been treating revocation (and OCSP) as a reliable mechanism. The major browsers all have out-of-band mechanisms for revoking known-malicious certs via something equivalent to the software update channel, which bypasses reliance on the CA infrastructure. If there's a large-scale attack, the relevant cert/CA will probably be disabled through that mechanism. And most of the smaller clients don't even bother with revocation checking at all (e.g., I'm pretty sure that on an average Linux system, things like curl or "import requests" do no revocation checking) so there's no point in undoing revocation if you're trying to attack one of those systems.
  - tialaramex 6y ago
    > e.g., I'm pretty sure that on an average Linux system, things like curl or "import requests" do no revocation checking This is correct. Even where there's some provision for checking, it's usually a mechanism where you can supply a CRL (Certificate Revocation List, a signed and dated document that says which certificates were revoked). CRLs are practical for a small private CA but they make no sense at scale. Let's Encrypt doesn't even have CRLs because they'd be enormous. To be fair 10-15 years ago there's a good chance that average Linux system has a set of CA roots which hasn't been updated in a decade, and most such clients aren't actually checking even CN let alone SANs so bad guys don't need a google.com certificate (or whatever) they can just get themselves a real certificate for actual-bad-guys.example and the client won't check the name matches anyway.
terom 6y ago
There's a reply from Ben Wilson (Mozilla) further down in the thread / the next day stating that Firefox as a client is not affected by the security issue (OCSP responses signed by these intermediate CAs would be rejected), and that Mozilla is not planning on enforcing the 7 day deadline for revocation of these intermediate certificates. The CAs will still need to replace these intermediate CAs, but with a more gradual timeline. https://www.mail-archive.com/dev-security-policy@lists.mozilla.org/msg13537.html https://www.mail-archive.com/dev-security-policy@lists.mozil... > We are concerned that revoking these impacted intermediate certificates within 7 days could cause more damage to the ecosystem than is warranted for this particular problem. Therefore, Mozilla does not plan to hold CAs to the BR requirement to revoke these certificates within 7 days. However, an additional Incident Report for delayed revocation will still be required, as per our documented process[2]. We want to work with CAs to identify a path forward, which includes determining a reasonable timeline and approach to replacing the certificates that incorrectly have the id-kp-OCSPSigning EKU (and performing key destruction for them).
- zenexer 6y ago
  That’s about what I expected. It doesn’t make sense for any parties involved to enforce the 7 day deadline, including users. This is a reasonable solution.
  - terom 6y ago
    It's just slightly disappointing that the CAs and browsers have agreed to operate under a policy that requires the CAs to revoke mis-issued subordinate CAs within 7 days (CA/B Forum BR 4.9.1.2), but that's not actually feasible in practice - at least not at this scale. Certificate issuance would need to be automated enough that it would be feasible to actually replace all certificates issued by the affected CAs within a 7-day period. Let's Encrypt represents the state of the art in terms of certificate automation, but last time they had an (impending) mass-revocation event, it turned out that even the ACME protocol / client implementations didn't really have any concept of an automated "this certificate is about to be revoked, please re-issue" process. As a result of that event, certbot at least got support for triggering renewal after revocation: https://github.com/certbot/certbot/issues/1028#issuecomment-593950500 https://github.com/certbot/certbot/issues/1028#issuecomment-... -> https://github.com/certbot/certbot/pull/7829 https://github.com/certbot/certbot/pull/7829

zenexer 6y ago

Flagging for a title change, as the revocation deadline was proposed by a single individual on a mailing list, and there’s no evidence from further discussion in that thread that a consensus has been reached. I would not expect all these certificates to be revoked within 7 days.

mbrinkers 6y ago
I changed the title. Although I heard from someone involved that the intermediates really should be revoked in 7 days. Let's wait and see.
- zenexer 6y ago
  They definitely should be—that’s what the author is claiming is mandated, and it would make sense. However, I’m a bit skeptical about browsers being able to enforce that timeline here. Also, given that the underlying cause appears to be ignorance, it would be prudent to take things slow and ensure that this doesn’t happen again. As I said before, the damage is already done—revoking appears to be insufficient here. If this does actually happen within 7 days, though, I will be thoroughly impressed.
  - floatingatoll 6y ago
    It could be considered a tacit warning that browsers may choose to mistrust the impacted subCAs in the near future. I don’t know the specifics, but I assume they can revoke for non-compliance using in-browser mechanisms without depending on the revocation process. EDIT: Mozilla’s reply: https://news.ycombinator.com/item?id=23748561 https://news.ycombinator.com/item?id=23748561
  - tptacek 6y ago
    You don't have to trust Sleevi (though: you always should); you can just read the BRs. The revocation requirement is in this case black-letter SHALL. https://cabforum.org/wp-content/uploads/CA-Browser-Forum-BR-1.7.0-1.pdf https://cabforum.org/wp-content/uploads/CA-Browser-Forum-BR-...
EE84M3i 6y ago
That "single individual on a mailing list" is Ryan Sleevi, who is basically the voice of Google in all matters to do with TLS PKI, and as I understand it is not working with a "opinions my own" cop-out in this context. It's not some rando

neallindsay 6y ago

"...revocation of the affected certificates is insufficient... Instead, it must be proven that all copies of the keys have been destroyed." Isn't the main reason you would want to revoke keys because they were disclosed, making it impossible to destroy all copies?

jakear 6y ago
Further, if I can destroy all keys do I even need to revoke the cert? (Honest question, I’m actually not sure)
- floatingatoll 6y ago
  Revocation is required by policy, so the question is technically moot. It’s generally good practice to generate and publish a revocation prior to destroying a private key, though. To provide an analogy in the context of PGP keys, if an attacker somehow finds a backup of your revoked and destroyed private key someday, they will have trouble using it because your revocation will be public and on record.
floatingatoll 6y ago
Normally, yes. This is not a normal circumstance though. In this scenario, the misissued intermediates effectively have sudo access to cancel a revocation issued by the parent CA. This is equivalent to being told that a non-root user could cancel a userdel command run by root. For policy compliance, the intermediates have to revoke their certificates — but since the intermediates can immediately un-revoke themselves, proof of key destruction is necessary as well to ensure that they cannot.

gruez 6y ago

>The author goes on to explain that revocation of the affected certificates is insufficient, because they could be used to effectively reverse their own revocation at any point in the future. Instead, it must be proven that all copies of the keys have been destroyed. That’s quite an undertaking. How would this be verified? Presumably the keys are stored on HSMs, but you can I'm not sure how you can prove that you didn't make a backup of the key.

jchrisa 6y ago
The point is that you can't verify it. Sometimes HN humor is too dry for its own good.
michaelt 6y ago
In principle, you would hope a CA would be able to precisely account for the number of backup copies of their private key. In practice, of course, that doesn't mean every one of them will have done. There's 293 of them, after all.
floatingatoll 6y ago
It’s impossible to prevent a truly dedicated malicious actor from doing so, but enforcement through both policy and independent auditors — and quality of response to security incidents — provide several layers of defenses against this scenario. (As with all things, a perfect defense is ultimately impossible, but they put in a lot of effort to get a lot of nines of certainty.)
TrueDuality 6y ago
It is largely impossible to fully prove. CAs are supposed to keep detailed records of any issuing keys and what was called for was specifically "witnessed Key Destruction Reports" which involves third party independent confirmation of destruction of documented keys. In the event that a key with a Key Destruction Report shows up again, the responsible party for that key will have shown unacceptable negligence and will potentially be subject to the exclusion of their keys as a valid certificate signer. A lot of these companies core businesses rely on remaining in a position to sign certificates so it is in their best interest to protect that privilege by following the documentation requirements, and properly destroy their keys. It's effectively a pretty good stick.

xZuki 6y ago

This is all confusing to me but I've been having certificate issues today and it seems like this could be related. It's a weird coincidence if not! Basically on Chrome one of my sites is throwing: "NETT::ERR_CERTIFICATE_TRANSPARENCY_REQUIRED" for most users but not all, even though they're all on Chrome. It seems to work fine in other browsers. https://transparencyreport.google.com/https/certificates https://transparencyreport.google.com/https/certificates When I check my domain here it seems like I have got the transparency certificate so I shouldn't be getting this error. Is this related to what you're talking about? I would really appreciate any help. I'm using https://letsencrypt.org/ https://letsencrypt.org/ for the cert.

geofft 6y ago
Does your cert have an SCT? It would be strange for a Let's Encrypt cert to be missing it but certainly possible. Try running (replace both example.coms with your domain name) openssl s_client -connect example.com:443 -servername example.com </dev/null | openssl x509 -noout -text which should print an SCT extension at the end - my version displays it by numeric identifier "1.3.6.1.4.1.11129.2.4.2" but maybe newer versions display it by name. Alternatively, I think you might able to go to https://www.ssllabs.com/ssltest/ https://www.ssllabs.com/ssltest/ and see if your cert has "Certificate Transparency: Yes", but I'm not sure exactly what that means. Anyway, I don't think this is related, the question at hand is about OCSP, which is a different mechanism from Certificate Transparency. (Arguably Certificate Transparency is a replacement for revocation in general being flawed in practice for many reasons, but they're different mechanisms.)
floatingatoll 6y ago
That is probably not related. If you run SSLLabs against your host name, does it say “Certificate Transparency: Yes” or No? https://www.ssllabs.com/ssltest/analyze.html?d=your-hostname https://www.ssllabs.com/ssltest/analyze.html?d=your-hostname
- xZuki 6y ago
  Thanks. It says "yes".

btown 6y ago

What are the state-actor-level attack implications of this? Before this was revealed, a party that compromised (or was able to be issued) a certificate for a website could be reasonably likely to be detected and have that certificate revoked if they used it for large-scale MITM or redirection. But now, if the actor were to also compromise any one of these sub-CAs before the key was deleted, could they permanently be in a position to be able to unilaterally reverse any such revocations, effectively giving them carte blanche to begin a campaign of compromising websites in earnest with the knowledge that their attacks would now be "sticky?" What would the recourse be here if one of those keys were to be compromised, or even if there was reason to believe one might have been? Would the entire CA-level trust chain need to be distrusted, requiring re-issuance of all certificates on that chain?

geofft 6y ago
Yes and almost-yes - you don't need to destroy the root CA itself, but you need to physically destroy the keys for any affected subordinate CAs. (Revoking them isn't sufficient, because they can just un-revoke themselves.) From the post: "... the degree of this issue likely also justifies requiring witnessed Key Destruction Reports, in order to preserve the integrity of the issuer of these certificates (which may include the CA's root)." The "good" news is that most people haven't really been treating revocation (and OCSP) as a reliable mechanism. The major browsers all have out-of-band mechanisms for revoking known-malicious certs via something equivalent to the software update channel, which bypasses reliance on the CA infrastructure. If there's a large-scale attack, the relevant cert/CA will probably be disabled through that mechanism. And most of the smaller clients don't even bother with revocation checking at all (e.g., I'm pretty sure that on an average Linux system, things like curl or "import requests" do no revocation checking) so there's no point in undoing revocation if you're trying to attack one of those systems.
- tialaramex 6y ago
  > e.g., I'm pretty sure that on an average Linux system, things like curl or "import requests" do no revocation checking This is correct. Even where there's some provision for checking, it's usually a mechanism where you can supply a CRL (Certificate Revocation List, a signed and dated document that says which certificates were revoked). CRLs are practical for a small private CA but they make no sense at scale. Let's Encrypt doesn't even have CRLs because they'd be enormous. To be fair 10-15 years ago there's a good chance that average Linux system has a set of CA roots which hasn't been updated in a decade, and most such clients aren't actually checking even CN let alone SANs so bad guys don't need a google.com certificate (or whatever) they can just get themselves a real certificate for actual-bad-guys.example and the client won't check the name matches anyway.

terom 6y ago

There's a reply from Ben Wilson (Mozilla) further down in the thread / the next day stating that Firefox as a client is not affected by the security issue (OCSP responses signed by these intermediate CAs would be rejected), and that Mozilla is not planning on enforcing the 7 day deadline for revocation of these intermediate certificates. The CAs will still need to replace these intermediate CAs, but with a more gradual timeline. https://www.mail-archive.com/dev-security-policy@lists.mozilla.org/msg13537.html https://www.mail-archive.com/dev-security-policy@lists.mozil... > We are concerned that revoking these impacted intermediate certificates within 7 days could cause more damage to the ecosystem than is warranted for this particular problem. Therefore, Mozilla does not plan to hold CAs to the BR requirement to revoke these certificates within 7 days. However, an additional Incident Report for delayed revocation will still be required, as per our documented process[2]. We want to work with CAs to identify a path forward, which includes determining a reasonable timeline and approach to replacing the certificates that incorrectly have the id-kp-OCSPSigning EKU (and performing key destruction for them).

zenexer 6y ago
That’s about what I expected. It doesn’t make sense for any parties involved to enforce the 7 day deadline, including users. This is a reasonable solution.
- terom 6y ago
  It's just slightly disappointing that the CAs and browsers have agreed to operate under a policy that requires the CAs to revoke mis-issued subordinate CAs within 7 days (CA/B Forum BR 4.9.1.2), but that's not actually feasible in practice - at least not at this scale. Certificate issuance would need to be automated enough that it would be feasible to actually replace all certificates issued by the affected CAs within a 7-day period. Let's Encrypt represents the state of the art in terms of certificate automation, but last time they had an (impending) mass-revocation event, it turned out that even the ACME protocol / client implementations didn't really have any concept of an automated "this certificate is about to be revoked, please re-issue" process. As a result of that event, certbot at least got support for triggering renewal after revocation: https://github.com/certbot/certbot/issues/1028#issuecomment-593950500 https://github.com/certbot/certbot/issues/1028#issuecomment-... -> https://github.com/certbot/certbot/pull/7829 https://github.com/certbot/certbot/pull/7829