4 ms·

Sorry I guess I only know how to implement/use an SSL certificate not necessarily how the generation/providing part works. What did this outage mean? If you alr

by ge96 9y ago

Sorry I guess I only know how to implement/use an SSL certificate not necessarily how the generation/providing part works. What did this outage mean? If you already had a certificate generated, were you not affected by this? Or does a certificate enabled on a website need to be checked/validated. I did look up what OCSP is after looking at the article: >an Internet protocol used for obtaining the revocation status of an X.509 digital certificate. So does this mean they weren't able to verify that the SSL was still good and so you'd get a warning in the browser saying "This site is not secure" or something?

minhajuddin 9y ago
This outage mainly effected the certificate generation and renewal. People who wanted to renew an existing/expired SSL cert or wanted to generate a new SSL cert weren't able to do it during this window. However, I believe the impact was small enough. SSL certs don't need any external service for verification. The clients (in this case your browser) would have a set of root certificates issues by many CAs which serve as way for verifying. The impact wasn't huge as most people who do renewal had a one time failure in the cron jobs which renew the certs. And since the renewals are tried 1 month before cert expiry, this was a non issue.
- ge96 9y ago
  Oh that's a good point, I thought a week of notice in advance but a month sounds safer. It's still 90 days right? Thanks
  - minhajuddin 9y ago
    Yeah. The common advise is to have an automated setup for renewals so you don't have to worry about it. It is also very trivial. Just add `@weekly letsencrypt renew` to your cron job and you are done.
    - ge96 9y ago
      I know that it should be trivial. But on my local/vps servers the cron jobs would not execute. I wasn't sure why... on my raspberry pi however they executed as expected. edit: it's possible it's a root problem, as Raspberry Pi I don't believe can be root. I know I'm bad using root user.
ZoFreX 9y ago
(this is purely about the OCSP server outage, others have commented on the issuance server outage) In most cases, clients ignore any failure to contact the OCSP servers. This means that: 1) OCSP servers aren't an additional point of failure for your website 2) A man-in-the-middle attack using a stolen and revoked certificate can prevent your browser from knowing it's stolen by blocking the connection to the OCSP server Possibly due to #2, or other reasons, I can only speculate, some clients treat failure to contact OCSP servers more seriously and abort the connection. During the outage, those clients were unable to talk to servers that: 1) were using LetsEncrypt 2) enabled OCSP 3) did not have a valid stapled OCSP response (for example because OCSP stapling was not configured, or their server lost the response and couldn't get a new one during the downtime) The size of the intersection of affected clients and sites is very small, but during that window they were completely unable to talk to each other. So in broad terms the impact was very small, but for those affected it was quite large.
- tialaramex 9y ago
  As you observe, OCSP today is not widely respected (for most sites Chrome doesn't even check OCSP at all for example) which is bad news if anybody's certificate gets stolen or misused. OCSP Stapling is (part of) the eventual solution. Web server software will go get the OCSP answers for its own certificates, and "staple" those to the certificates when it serves them. So now client software doesn't have to wrestle with unreliable networks and make extra connections, the OCSP response is right there with the certificate during connection to the site. However, Quality of Implementation for OCSP Stapling in some of the most popular HTTP servers is poor. Let's take the example of Apache httpd, possibly still the most popular server in the world. By default Apache doesn't do OCSP stapling at all. So you need to configure that, doing so isn't even a one line "Yup, staple please" either, instead it appear the person writing this code for Apache went through the specification and any time they weren't sure what to do they said "Eh, I'll leave that to the sysadmin" and added a configuration option, with more or less random defaults. As a result by default Apache will forget a perfectly good OCSP answer it knows in favour of trying to get a new one. If that fails (as it did here due to Let's Encrypt's problems) Apache doesn't say "Oh well, I have a good one already, I can use that". It makes a fake "error" OCSP response and serves that up. Why? Nobody we've ever been able to find knows what that could be useful for, but the Apache developer decided it would be a good default. "Yup, if anything goes wrong, just irreversibly break the entire server, that way they'll be sure to notice". It will also happily staple worthless outdated answers, or answers saying e.g. "Temporary failure, try later", which likewise will just cause visitors to your site to get turned away, rather than continuing to use a known-good answer it has. And nobody at Apache seems the least bit interested in fixing this.

by ge96 9y ago

minhajuddin 9y ago

This outage mainly effected the certificate generation and renewal. People who wanted to renew an existing/expired SSL cert or wanted to generate a new SSL cert weren't able to do it during this window. However, I believe the impact was small enough. SSL certs don't need any external service for verification. The clients (in this case your browser) would have a set of root certificates issues by many CAs which serve as way for verifying. The impact wasn't huge as most people who do renewal had a one time failure in the cron jobs which renew the certs. And since the renewals are tried 1 month before cert expiry, this was a non issue.

ge96 9y ago
Oh that's a good point, I thought a week of notice in advance but a month sounds safer. It's still 90 days right? Thanks
- minhajuddin 9y ago
  Yeah. The common advise is to have an automated setup for renewals so you don't have to worry about it. It is also very trivial. Just add `@weekly letsencrypt renew` to your cron job and you are done.
  - ge96 9y ago
    I know that it should be trivial. But on my local/vps servers the cron jobs would not execute. I wasn't sure why... on my raspberry pi however they executed as expected. edit: it's possible it's a root problem, as Raspberry Pi I don't believe can be root. I know I'm bad using root user.

ZoFreX 9y ago

(this is purely about the OCSP server outage, others have commented on the issuance server outage) In most cases, clients ignore any failure to contact the OCSP servers. This means that: 1) OCSP servers aren't an additional point of failure for your website 2) A man-in-the-middle attack using a stolen and revoked certificate can prevent your browser from knowing it's stolen by blocking the connection to the OCSP server Possibly due to #2, or other reasons, I can only speculate, some clients treat failure to contact OCSP servers more seriously and abort the connection. During the outage, those clients were unable to talk to servers that: 1) were using LetsEncrypt 2) enabled OCSP 3) did not have a valid stapled OCSP response (for example because OCSP stapling was not configured, or their server lost the response and couldn't get a new one during the downtime) The size of the intersection of affected clients and sites is very small, but during that window they were completely unable to talk to each other. So in broad terms the impact was very small, but for those affected it was quite large.

tialaramex 9y ago
As you observe, OCSP today is not widely respected (for most sites Chrome doesn't even check OCSP at all for example) which is bad news if anybody's certificate gets stolen or misused. OCSP Stapling is (part of) the eventual solution. Web server software will go get the OCSP answers for its own certificates, and "staple" those to the certificates when it serves them. So now client software doesn't have to wrestle with unreliable networks and make extra connections, the OCSP response is right there with the certificate during connection to the site. However, Quality of Implementation for OCSP Stapling in some of the most popular HTTP servers is poor. Let's take the example of Apache httpd, possibly still the most popular server in the world. By default Apache doesn't do OCSP stapling at all. So you need to configure that, doing so isn't even a one line "Yup, staple please" either, instead it appear the person writing this code for Apache went through the specification and any time they weren't sure what to do they said "Eh, I'll leave that to the sysadmin" and added a configuration option, with more or less random defaults. As a result by default Apache will forget a perfectly good OCSP answer it knows in favour of trying to get a new one. If that fails (as it did here due to Let's Encrypt's problems) Apache doesn't say "Oh well, I have a good one already, I can use that". It makes a fake "error" OCSP response and serves that up. Why? Nobody we've ever been able to find knows what that could be useful for, but the Apache developer decided it would be a good default. "Yup, if anything goes wrong, just irreversibly break the entire server, that way they'll be sure to notice". It will also happily staple worthless outdated answers, or answers saying e.g. "Temporary failure, try later", which likewise will just cause visitors to your site to get turned away, rather than continuing to use a known-good answer it has. And nobody at Apache seems the least bit interested in fixing this.