6 ms·

Let's Encrypt OCSP and Issuance Outage Postmortem

223 points by agrajag 9y ago

zeta0134 9y ago
And this is why you always... aww hell, who am I kidding? That was one complex series of events, and I would have done no better. Grats to the Let's Encrypt team for figuring it out, and thanks for writing up the post mortem. It's always interesting to read things like this, and it just goes to show that sometimes all the monitoring you thought you had covered wasn't quite enough.
- zzzcpan 9y ago
  It's easy to do better if you ever listened to Joe Armstrong or read his dissertation. The idea of centralized monitoring is just broken and can never be truly reliable.
  - sbov 9y ago
    How does Joe Armstrong's dissertation help you monitor the correct metrics? Edit: or what about his dissertation would have made monitoring unnecessary?
    - cat199 9y ago
      It's easy to do better if you've ever armchair quarterbacked or just read about doing it. The idea of actually outlining a coherent argument is just broken and can never be truly reliable.
  - bogomipz 9y ago
    Can you say who is Joe Armstrong and what is the dissertation? I would be interested in reading it.
    - krallja 9y ago
      Joe Armstrong's dissertation became Erlang
      - bogomipz 9y ago
        Oh wow I didn't know any of this. This will be some good weekend reading. Thanks. Here is the link in case anyone else is interested: http://erlang.org/download/armstrong_thesis_2003.pdf http://erlang.org/download/armstrong_thesis_2003.pdf
        arianvanp 9y ago
        Anybody know what LaTeX book template this is?
        SkyMarshal 9y ago
        Their original video demo of hot code swapping is also a classic worth checking out: https://www.youtube.com/watch?v=uKfKtXYLG78 https://www.youtube.com/watch?v=uKfKtXYLG78 That got a lot of programmers interested in Erlang ~10yrs ago.
      - krallja 9y ago
        I got the order backwards: Erlang (1981) became Joe Armstrong's dissertation (2003).
    - deleted 9y ago
      [deleted]
- donalhunt 9y ago
  I'm not sure (at least on the rollout side which was what caused the largest impact)... My team does a lot of releases and rollouts. We use automation in the majority of cases and automatically pause if alert conditions occur during the rollout. We usually release a dev version (which we can run tests (incl. typical traffic that we expect the service to handle)), canary version and production version. Ideally you want to catch this sort of stuff during the tests that you run against your dev instances. Of course, you have to have the right testing infrastructure, monitoring and alerting in place to avail of that. For some of our services, we use graduated rollouts so we can gradually direct traffic from the old version of the stack to the new one over a period of time. This allows us (where us = our monitoring) to quickly identify if there are issues (e.g. can compare error rates of version x with version x+1) and reset the percentage of traffic going to version x+1 to 0% if there is an issue. From the postmortem, they indicated the need to add additional monitoring to identify an increase in error rate + traffic rate. I'm pretty sure the Production Readiness Reviews called out in the SRE book by Google (see here: https://landing.google.com/sre/book/chapters/evolving-sre-engagement-model.html https://landing.google.com/sre/book/chapters/evolving-sre-en...) would have caught this. The book also has some good insights on how to do better monitoring (hint: don't have humans eyeballing dashboards). Saying all that, I've seen plenty of small bugs make it to that stage of deployment with that level of impact so it's certainly not trivial to get 100% test / monitoring coverage... Reviewing the causes of outages and specifically identifying and resolving action items from them go a long way to ensuring you improve over time.
- baby 9y ago
  So they were sending b64 strings in URLs? Of course this would fail :|
  - corobo 9y ago
    Why's that?
    - baby 9y ago
      Because of the way slashes (a b64 char) gets parsed in URLs.
yuhong 9y ago
I have been thinking of a yellow single click SSL warning for thing like OCSP server cannot be contacted. Not enabled by default for now, of course.
agrajag 9y ago
I thought this was a great writeup, but one significant issue I see unanswered is why did all OCSP responses fall out of cache at the same time? In a cache with an even distribution of expiration times they should have seen a gradual increase in traffic as responses steadily fell out of cache. Adding jitter when setting cache duration for the CDNshould help even out the rate at which responses fall out of cache. In addition, monitoring cache rate and measuring request rate at the CDN should have been big indicators that it wasn't a DDoS. Lastly, is this kind of upstream throttling with no customer communication common? That seems like a big failing on the ISPs side.
- zeta0134 9y ago
  I work at a hosting company, and I can attest to throttling at the ISP side being somewhat common. My own company will throttle or null route customers almost immediately if enough traffic comes through that it starts to affect other customers behind the same switches. We try to notify customers as quickly as possible (emails go out within minutes) but there are a lot of cases where the emails end up in unmonitored inboxes and customers don't realize they're down until their clients complain to them about it. In any case, it sounds like there may be a bit of a logic bug clientside if a service downtime causes all of their clients to generate so much traffic that it looks like DDoS. The clients should be throttling themselves to prevent overloading a downed upstream service, why didn't that happen here? That's worth investigating, though much more difficult to fix at this point with that many copies of the client in the wild. EDIT: I just thought to actually look up OCSP, I assumed it was the mechanism something like certbot was using to renew certificated. This is built into browsers? Yeah, nevermind on this whole paragraph then.
  - mholt 9y ago
    > This is built into browsers? Yup, unfortunately, browsers often have to do revocation (and SCT) checks because traditional servers like Apache and nginx don't staple OCSP responses by default (and even if they are configured to do so, the implementations in these servers are not robust against outages). Firefox checks OCSP for DV certs, but that will be disabled in a near-future release.
theprop 9y ago
LetsEncrypt is just awesome & a superb mission to make every website easily secured!! SSLs were otherwise a big scam! (Yes i know I love them!)
ge96 9y ago
I used Let's Encrypt for the first time today, awesome! Saved $9.00 haha
ge96 9y ago
Sorry I guess I only know how to implement/use an SSL certificate not necessarily how the generation/providing part works. What did this outage mean? If you already had a certificate generated, were you not affected by this? Or does a certificate enabled on a website need to be checked/validated. I did look up what OCSP is after looking at the article: >an Internet protocol used for obtaining the revocation status of an X.509 digital certificate. So does this mean they weren't able to verify that the SSL was still good and so you'd get a warning in the browser saying "This site is not secure" or something?
- minhajuddin 9y ago
  This outage mainly effected the certificate generation and renewal. People who wanted to renew an existing/expired SSL cert or wanted to generate a new SSL cert weren't able to do it during this window. However, I believe the impact was small enough. SSL certs don't need any external service for verification. The clients (in this case your browser) would have a set of root certificates issues by many CAs which serve as way for verifying. The impact wasn't huge as most people who do renewal had a one time failure in the cron jobs which renew the certs. And since the renewals are tried 1 month before cert expiry, this was a non issue.
  - ge96 9y ago
    Oh that's a good point, I thought a week of notice in advance but a month sounds safer. It's still 90 days right? Thanks
    - minhajuddin 9y ago
      Yeah. The common advise is to have an automated setup for renewals so you don't have to worry about it. It is also very trivial. Just add `@weekly letsencrypt renew` to your cron job and you are done.
      - ge96 9y ago
        I know that it should be trivial. But on my local/vps servers the cron jobs would not execute. I wasn't sure why... on my raspberry pi however they executed as expected. edit: it's possible it's a root problem, as Raspberry Pi I don't believe can be root. I know I'm bad using root user.
Already__Taken 9y ago
Is it possible to have a test stage that replays/simulates the entire previous 24 hours of production to compare previous output with new output. This might let you know if a fix to a known bug actually makes things worse. Obviously I'm not inferring this is a step mozilla should already have in place.
- woliveirajr 9y ago
  That's a huge step. Lots of data to save for the input, lots of diff output, and you have to look for all details of the output, since many of them are desirable. Then you would automate the output processing, and a wrong rule would miss the new bug (because it was unexpected, so the automation would account for it).

223 points by agrajag 9y ago

And this is why you always... aww hell, who am I kidding? That was one complex series of events, and I would have done no better. Grats to the Let's Encrypt team for figuring it out, and thanks for writing up the post mortem. It's always interesting to read things like this, and it just goes to show that sometimes all the monitoring you thought you had covered wasn't quite enough.

zzzcpan 9y ago
It's easy to do better if you ever listened to Joe Armstrong or read his dissertation. The idea of centralized monitoring is just broken and can never be truly reliable.
- sbov 9y ago
  How does Joe Armstrong's dissertation help you monitor the correct metrics? Edit: or what about his dissertation would have made monitoring unnecessary?
  - cat199 9y ago
    It's easy to do better if you've ever armchair quarterbacked or just read about doing it. The idea of actually outlining a coherent argument is just broken and can never be truly reliable.
- bogomipz 9y ago
  Can you say who is Joe Armstrong and what is the dissertation? I would be interested in reading it.
  - krallja 9y ago
    Joe Armstrong's dissertation became Erlang
    - bogomipz 9y ago
      Oh wow I didn't know any of this. This will be some good weekend reading. Thanks. Here is the link in case anyone else is interested: http://erlang.org/download/armstrong_thesis_2003.pdf http://erlang.org/download/armstrong_thesis_2003.pdf
      - arianvanp 9y ago
        Anybody know what LaTeX book template this is?
      - SkyMarshal 9y ago
        Their original video demo of hot code swapping is also a classic worth checking out: https://www.youtube.com/watch?v=uKfKtXYLG78 https://www.youtube.com/watch?v=uKfKtXYLG78 That got a lot of programmers interested in Erlang ~10yrs ago.
    - krallja 9y ago
      I got the order backwards: Erlang (1981) became Joe Armstrong's dissertation (2003).
  - deleted 9y ago
    [deleted]
donalhunt 9y ago
I'm not sure (at least on the rollout side which was what caused the largest impact)... My team does a lot of releases and rollouts. We use automation in the majority of cases and automatically pause if alert conditions occur during the rollout. We usually release a dev version (which we can run tests (incl. typical traffic that we expect the service to handle)), canary version and production version. Ideally you want to catch this sort of stuff during the tests that you run against your dev instances. Of course, you have to have the right testing infrastructure, monitoring and alerting in place to avail of that. For some of our services, we use graduated rollouts so we can gradually direct traffic from the old version of the stack to the new one over a period of time. This allows us (where us = our monitoring) to quickly identify if there are issues (e.g. can compare error rates of version x with version x+1) and reset the percentage of traffic going to version x+1 to 0% if there is an issue. From the postmortem, they indicated the need to add additional monitoring to identify an increase in error rate + traffic rate. I'm pretty sure the Production Readiness Reviews called out in the SRE book by Google (see here: https://landing.google.com/sre/book/chapters/evolving-sre-engagement-model.html https://landing.google.com/sre/book/chapters/evolving-sre-en...) would have caught this. The book also has some good insights on how to do better monitoring (hint: don't have humans eyeballing dashboards). Saying all that, I've seen plenty of small bugs make it to that stage of deployment with that level of impact so it's certainly not trivial to get 100% test / monitoring coverage... Reviewing the causes of outages and specifically identifying and resolving action items from them go a long way to ensuring you improve over time.
baby 9y ago
So they were sending b64 strings in URLs? Of course this would fail :|
- corobo 9y ago
  Why's that?
  - baby 9y ago
    Because of the way slashes (a b64 char) gets parsed in URLs.

yuhong 9y ago

I have been thinking of a yellow single click SSL warning for thing like OCSP server cannot be contacted. Not enabled by default for now, of course.

agrajag 9y ago

I thought this was a great writeup, but one significant issue I see unanswered is why did all OCSP responses fall out of cache at the same time? In a cache with an even distribution of expiration times they should have seen a gradual increase in traffic as responses steadily fell out of cache. Adding jitter when setting cache duration for the CDNshould help even out the rate at which responses fall out of cache. In addition, monitoring cache rate and measuring request rate at the CDN should have been big indicators that it wasn't a DDoS. Lastly, is this kind of upstream throttling with no customer communication common? That seems like a big failing on the ISPs side.

zeta0134 9y ago
I work at a hosting company, and I can attest to throttling at the ISP side being somewhat common. My own company will throttle or null route customers almost immediately if enough traffic comes through that it starts to affect other customers behind the same switches. We try to notify customers as quickly as possible (emails go out within minutes) but there are a lot of cases where the emails end up in unmonitored inboxes and customers don't realize they're down until their clients complain to them about it. In any case, it sounds like there may be a bit of a logic bug clientside if a service downtime causes all of their clients to generate so much traffic that it looks like DDoS. The clients should be throttling themselves to prevent overloading a downed upstream service, why didn't that happen here? That's worth investigating, though much more difficult to fix at this point with that many copies of the client in the wild. EDIT: I just thought to actually look up OCSP, I assumed it was the mechanism something like certbot was using to renew certificated. This is built into browsers? Yeah, nevermind on this whole paragraph then.
- mholt 9y ago
  > This is built into browsers? Yup, unfortunately, browsers often have to do revocation (and SCT) checks because traditional servers like Apache and nginx don't staple OCSP responses by default (and even if they are configured to do so, the implementations in these servers are not robust against outages). Firefox checks OCSP for DV certs, but that will be disabled in a near-future release.

theprop 9y ago

LetsEncrypt is just awesome & a superb mission to make every website easily secured!! SSLs were otherwise a big scam! (Yes i know I love them!)

ge96 9y ago

I used Let's Encrypt for the first time today, awesome! Saved $9.00 haha

ge96 9y ago

Sorry I guess I only know how to implement/use an SSL certificate not necessarily how the generation/providing part works. What did this outage mean? If you already had a certificate generated, were you not affected by this? Or does a certificate enabled on a website need to be checked/validated. I did look up what OCSP is after looking at the article: >an Internet protocol used for obtaining the revocation status of an X.509 digital certificate. So does this mean they weren't able to verify that the SSL was still good and so you'd get a warning in the browser saying "This site is not secure" or something?

minhajuddin 9y ago
This outage mainly effected the certificate generation and renewal. People who wanted to renew an existing/expired SSL cert or wanted to generate a new SSL cert weren't able to do it during this window. However, I believe the impact was small enough. SSL certs don't need any external service for verification. The clients (in this case your browser) would have a set of root certificates issues by many CAs which serve as way for verifying. The impact wasn't huge as most people who do renewal had a one time failure in the cron jobs which renew the certs. And since the renewals are tried 1 month before cert expiry, this was a non issue.
- ge96 9y ago
  Oh that's a good point, I thought a week of notice in advance but a month sounds safer. It's still 90 days right? Thanks
  - minhajuddin 9y ago
    Yeah. The common advise is to have an automated setup for renewals so you don't have to worry about it. It is also very trivial. Just add `@weekly letsencrypt renew` to your cron job and you are done.
    - ge96 9y ago
      I know that it should be trivial. But on my local/vps servers the cron jobs would not execute. I wasn't sure why... on my raspberry pi however they executed as expected. edit: it's possible it's a root problem, as Raspberry Pi I don't believe can be root. I know I'm bad using root user.

Already__Taken 9y ago

Is it possible to have a test stage that replays/simulates the entire previous 24 hours of production to compare previous output with new output. This might let you know if a fix to a known bug actually makes things worse. Obviously I'm not inferring this is a step mozilla should already have in place.

woliveirajr 9y ago
That's a huge step. Lots of data to save for the input, lots of diff output, and you have to look for all details of the output, since many of them are desirable. Then you would automate the output processing, and a wrong rule would miss the new bug (because it was unexpected, so the automation would account for it).