6 ms·
And this is why you always... aww hell, who am I kidding? That was one complex series of events, and I would have done no better. Grats to the Let's Encrypt te
by zeta0134 9y ago
And this is why you always... aww hell, who am I kidding? That was one complex series of events, and I would have done no better.
Grats to the Let's Encrypt team for figuring it out, and thanks for writing up the post mortem. It's always interesting to read things like this, and it just goes to show that sometimes all the monitoring you thought you had covered wasn't quite enough.
- zzzcpan 9y agoIt's easy to do better if you ever listened to Joe Armstrong or read his dissertation. The idea of centralized monitoring is just broken and can never be truly reliable.
- sbov 9y agoHow does Joe Armstrong's dissertation help you monitor the correct metrics? Edit: or what about his dissertation would have made monitoring unnecessary?
- cat199 9y agoIt's easy to do better if you've ever armchair quarterbacked or just read about doing it. The idea of actually outlining a coherent argument is just broken and can never be truly reliable.
- bogomipz 9y agoCan you say who is Joe Armstrong and what is the dissertation? I would be interested in reading it.
- krallja 9y agoJoe Armstrong's dissertation became Erlang
- bogomipz 9y agoOh wow I didn't know any of this. This will be some good weekend reading. Thanks. Here is the link in case anyone else is interested: http://erlang.org/download/armstrong_thesis_2003.pdf http://erlang.org/download/armstrong_thesis_2003.pdf
- arianvanp 9y agoAnybody know what LaTeX book template this is?
- SkyMarshal 9y agoTheir original video demo of hot code swapping is also a classic worth checking out: https://www.youtube.com/watch?v=uKfKtXYLG78 https://www.youtube.com/watch?v=uKfKtXYLG78 That got a lot of programmers interested in Erlang ~10yrs ago.
- krallja 9y agoI got the order backwards: Erlang (1981) became Joe Armstrong's dissertation (2003).
- deleted 9y ago[deleted]
- donalhunt 9y agoI'm not sure (at least on the rollout side which was what caused the largest impact)... My team does a lot of releases and rollouts. We use automation in the majority of cases and automatically pause if alert conditions occur during the rollout. We usually release a dev version (which we can run tests (incl. typical traffic that we expect the service to handle)), canary version and production version. Ideally you want to catch this sort of stuff during the tests that you run against your dev instances. Of course, you have to have the right testing infrastructure, monitoring and alerting in place to avail of that. For some of our services, we use graduated rollouts so we can gradually direct traffic from the old version of the stack to the new one over a period of time. This allows us (where us = our monitoring) to quickly identify if there are issues (e.g. can compare error rates of version x with version x+1) and reset the percentage of traffic going to version x+1 to 0% if there is an issue. From the postmortem, they indicated the need to add additional monitoring to identify an increase in error rate + traffic rate. I'm pretty sure the Production Readiness Reviews called out in the SRE book by Google (see here: https://landing.google.com/sre/book/chapters/evolving-sre-engagement-model.html https://landing.google.com/sre/book/chapters/evolving-sre-en...) would have caught this. The book also has some good insights on how to do better monitoring (hint: don't have humans eyeballing dashboards). Saying all that, I've seen plenty of small bugs make it to that stage of deployment with that level of impact so it's certainly not trivial to get 100% test / monitoring coverage... Reviewing the causes of outages and specifically identifying and resolving action items from them go a long way to ensuring you improve over time.
- baby 9y agoSo they were sending b64 strings in URLs? Of course this would fail :|