4 ms·
"It happens" is the only reasonable takeaway you can get from a postmortem like this. My worry is that people read it and go "I am aghast that such a command ca
by cflewis 10y ago
"It happens" is the only reasonable takeaway you can get from a postmortem like this. My worry is that people read it and go "I am aghast that such a command can be run!" without knowing that little commands like that are run numerous times a day without incident.
The only thing I read in there and go "hmmm" is that it took quite that long for the S3 service to recover, and that the status page wasn't hosted on someone that doesn't have an S3 dependency. That's just a plain "doh" moment :)
- ProAm 10y agoPeople need to realize when they go to the cloud it's not that 'it happens', it's that it will happen, and you have no ability to do anything about it. Fact of life and risk management.
- fixermark 10y ago... and it's a different risk from self-hosting, but self-hosting provides all sorts of similar issues (such as when you do this to yourself, the cost is now coming out of your pocket, not Amazon's, to employ software engineers to harden your scripts against making the same mistake twice).
- thr0waway1239 10y agoNot to mention, Amazon is catching the long tail of cloud failure like Google is catching the long tail of search keywords. They can now say with a somewhat straight face - "You know all those scripts you run to keep everything up? We have figured out many, many more possible ways for them to fail than you probably ever will, and we have added more layers of safeguards than you can even imagine."