4 ms·

(this is purely about the OCSP server outage, others have commented on the issuance server outage) In most cases, clients ignore any failure to contact the OCS

by ZoFreX 9y ago

(this is purely about the OCSP server outage, others have commented on the issuance server outage) In most cases, clients ignore any failure to contact the OCSP servers. This means that: 1) OCSP servers aren't an additional point of failure for your website 2) A man-in-the-middle attack using a stolen and revoked certificate can prevent your browser from knowing it's stolen by blocking the connection to the OCSP server Possibly due to #2, or other reasons, I can only speculate, some clients treat failure to contact OCSP servers more seriously and abort the connection. During the outage, those clients were unable to talk to servers that: 1) were using LetsEncrypt 2) enabled OCSP 3) did not have a valid stapled OCSP response (for example because OCSP stapling was not configured, or their server lost the response and couldn't get a new one during the downtime) The size of the intersection of affected clients and sites is very small, but during that window they were completely unable to talk to each other. So in broad terms the impact was very small, but for those affected it was quite large.

tialaramex 9y ago
As you observe, OCSP today is not widely respected (for most sites Chrome doesn't even check OCSP at all for example) which is bad news if anybody's certificate gets stolen or misused. OCSP Stapling is (part of) the eventual solution. Web server software will go get the OCSP answers for its own certificates, and "staple" those to the certificates when it serves them. So now client software doesn't have to wrestle with unreliable networks and make extra connections, the OCSP response is right there with the certificate during connection to the site. However, Quality of Implementation for OCSP Stapling in some of the most popular HTTP servers is poor. Let's take the example of Apache httpd, possibly still the most popular server in the world. By default Apache doesn't do OCSP stapling at all. So you need to configure that, doing so isn't even a one line "Yup, staple please" either, instead it appear the person writing this code for Apache went through the specification and any time they weren't sure what to do they said "Eh, I'll leave that to the sysadmin" and added a configuration option, with more or less random defaults. As a result by default Apache will forget a perfectly good OCSP answer it knows in favour of trying to get a new one. If that fails (as it did here due to Let's Encrypt's problems) Apache doesn't say "Oh well, I have a good one already, I can use that". It makes a fake "error" OCSP response and serves that up. Why? Nobody we've ever been able to find knows what that could be useful for, but the Apache developer decided it would be a good default. "Yup, if anything goes wrong, just irreversibly break the entire server, that way they'll be sure to notice". It will also happily staple worthless outdated answers, or answers saying e.g. "Temporary failure, try later", which likewise will just cause visitors to your site to get turned away, rather than continuing to use a known-good answer it has. And nobody at Apache seems the least bit interested in fixing this.
- ge96 9y ago
  Man I do not follow much... part of it. I use Apache myself. I followed Digital Ocean's guide on how to setup Let's Encrypt and I was concerned about the mention of "backports" I didn't really get that. I've setup SSL certificates before regarding the Virtual hosts/ got some help with proper cipher suites and what not (have an A+ rating on qualyss) but yeah... after getting over the .pem vs. .cert it was no problem... though sometimes the process would say "no domain found" or "more than one virtual host" when there was only one. At any rate it worked for me so I'm glad I finally took the plunge.
- Anthony-G 9y ago
  As an Apache user, I’ve yet to look into enabling OCSP stapling so thanks for this informative post. I presume the developer you are referring to is (one of) the developers mod_ssl. I found the bug report[1] where the Apache developers state that they won’t enable stapling by default because “it would enable a "phoning home" feature (to the CA's OCSP responders) as a side effect of configuring a certificate”. That seems reasonable to me. However, the other behaviour that you’ve mentioned seems less so. Do you have any references (mailing list discussions, links to bug reports, etc.) for this? By the way, your opening line should probably be edited to say something like which is bad news if anybody's private key gets stolen or misused and they need to revoke the corresponding key(s). Most readers of this discussion will know what you mean but some who are still learning about PKI may be confused. [1] https://bz.apache.org/bugzilla/show_bug.cgi?id=50740#c20 https://bz.apache.org/bugzilla/show_bug.cgi?id=50740#c20
  - Anthony-G 9y ago
    Wikipedia has a good description on what OCSP Stapling is[1] and how it works. When I read the Apache projects' WONTFIX reason, I presumed that it was related to how plain OCSP requires the client to "phone home" in order to check whether a certificate has been revoked or not – which has implications for the privacy of the browser. However, now that I know OCSP Stapling works (the web server caches and proxies time-stamped OCSP responses that are signed by the CA), the Apache position is much less reasonable. As a Let’s Encrypt user, I “phone home” every couple of months to renew my X.509 key and certificate. That’s not a privacy concern for me or anyone else who happens to browse my site. I also found a good article by Hanno Böck[2] which provides more details on how OCSP Stapling is thoroughly broken on Apache as described by tialaramex). [1] https://en.wikipedia.org/wiki/OCSP_stapling https://en.wikipedia.org/wiki/OCSP_stapling [2] https://blog.hboeck.de/archives/886-The-Problem-with-OCSP-Stapling-and-Must-Staple-and-why-Certificate-Revocation-is-still-broken.html https://blog.hboeck.de/archives/886-The-Problem-with-OCSP-St...

by ZoFreX 9y ago

tialaramex 9y ago

As you observe, OCSP today is not widely respected (for most sites Chrome doesn't even check OCSP at all for example) which is bad news if anybody's certificate gets stolen or misused. OCSP Stapling is (part of) the eventual solution. Web server software will go get the OCSP answers for its own certificates, and "staple" those to the certificates when it serves them. So now client software doesn't have to wrestle with unreliable networks and make extra connections, the OCSP response is right there with the certificate during connection to the site. However, Quality of Implementation for OCSP Stapling in some of the most popular HTTP servers is poor. Let's take the example of Apache httpd, possibly still the most popular server in the world. By default Apache doesn't do OCSP stapling at all. So you need to configure that, doing so isn't even a one line "Yup, staple please" either, instead it appear the person writing this code for Apache went through the specification and any time they weren't sure what to do they said "Eh, I'll leave that to the sysadmin" and added a configuration option, with more or less random defaults. As a result by default Apache will forget a perfectly good OCSP answer it knows in favour of trying to get a new one. If that fails (as it did here due to Let's Encrypt's problems) Apache doesn't say "Oh well, I have a good one already, I can use that". It makes a fake "error" OCSP response and serves that up. Why? Nobody we've ever been able to find knows what that could be useful for, but the Apache developer decided it would be a good default. "Yup, if anything goes wrong, just irreversibly break the entire server, that way they'll be sure to notice". It will also happily staple worthless outdated answers, or answers saying e.g. "Temporary failure, try later", which likewise will just cause visitors to your site to get turned away, rather than continuing to use a known-good answer it has. And nobody at Apache seems the least bit interested in fixing this.

ge96 9y ago
Man I do not follow much... part of it. I use Apache myself. I followed Digital Ocean's guide on how to setup Let's Encrypt and I was concerned about the mention of "backports" I didn't really get that. I've setup SSL certificates before regarding the Virtual hosts/ got some help with proper cipher suites and what not (have an A+ rating on qualyss) but yeah... after getting over the .pem vs. .cert it was no problem... though sometimes the process would say "no domain found" or "more than one virtual host" when there was only one. At any rate it worked for me so I'm glad I finally took the plunge.
Anthony-G 9y ago
As an Apache user, I’ve yet to look into enabling OCSP stapling so thanks for this informative post. I presume the developer you are referring to is (one of) the developers mod_ssl. I found the bug report[1] where the Apache developers state that they won’t enable stapling by default because “it would enable a "phoning home" feature (to the CA's OCSP responders) as a side effect of configuring a certificate”. That seems reasonable to me. However, the other behaviour that you’ve mentioned seems less so. Do you have any references (mailing list discussions, links to bug reports, etc.) for this? By the way, your opening line should probably be edited to say something like which is bad news if anybody's private key gets stolen or misused and they need to revoke the corresponding key(s). Most readers of this discussion will know what you mean but some who are still learning about PKI may be confused. [1] https://bz.apache.org/bugzilla/show_bug.cgi?id=50740#c20 https://bz.apache.org/bugzilla/show_bug.cgi?id=50740#c20
- Anthony-G 9y ago
  Wikipedia has a good description on what OCSP Stapling is[1] and how it works. When I read the Apache projects' WONTFIX reason, I presumed that it was related to how plain OCSP requires the client to "phone home" in order to check whether a certificate has been revoked or not – which has implications for the privacy of the browser. However, now that I know OCSP Stapling works (the web server caches and proxies time-stamped OCSP responses that are signed by the CA), the Apache position is much less reasonable. As a Let’s Encrypt user, I “phone home” every couple of months to renew my X.509 key and certificate. That’s not a privacy concern for me or anyone else who happens to browse my site. I also found a good article by Hanno Böck[2] which provides more details on how OCSP Stapling is thoroughly broken on Apache as described by tialaramex). [1] https://en.wikipedia.org/wiki/OCSP_stapling https://en.wikipedia.org/wiki/OCSP_stapling [2] https://blog.hboeck.de/archives/886-The-Problem-with-OCSP-Stapling-and-Must-Staple-and-why-Certificate-Revocation-is-still-broken.html https://blog.hboeck.de/archives/886-The-Problem-with-OCSP-St...