3 ms·
Agreed, especially regarding the culture but isn't this pretty much the same explanation they gave a few years ago when something similar happened? I seem to r
by mulmen 10y ago
Agreed, especially regarding the culture but isn't this pretty much the same explanation they gave a few years ago when something similar happened?
I seem to recall an EC2 or S3 outage a few years ago that boiled down to an engineer pushing out a patch that broke an entire region when it was supposed to be a phased deployment.
I could be mis-remembering that but it's important that these lessons be applied across the whole company (at least AWS) so it would be a bigger mark against AWS if this is a result of similar tooling to what caused a previous outage.
- cjbprime 10y agoPretty sure that one was a Microsoft Azure outage. (Source: am a self-identified post-mortems connoisseur. :)
- evilDagmar 10y agoNot a bad plan. If you don't make enough mistakes on your own, ya gotta learn from the mistakes of others as a preventative.
- sah2ed 10y agoDo you by chance keep a public log of your postmortem collection :)?
- cjbprime 10y agoI don't, but danluu does! https://github.com/danluu/post-mortems https://github.com/danluu/post-mortems
- bobzimuta 10y agoYeah an EC2 engineer switched over traffic to a backup network connection that had significantly less bandwidth, triggering cascading failures.