4 ms·
Which is really the point of automation and configuration management. When a manager asks you, "How are you going to prevent this in the future?" You can say, "
by djsumdog 10y ago
Which is really the point of automation and configuration management. When a manager asks you, "How are you going to prevent this in the future?" You can say, "We added a check so n must be less than x% of the total number of cluster members," or "We added additional unit tests for the missing area of coverage" or "We added new integration tests that will pick up on this."
Tests and configuration scripts don't prevent all breakage. But when you have them, you can say, "We missed that, let's add it," or "That failed, but it's a false positive. Let's add this edge case to this test."
If you have no automation, tests or auditing systems around running deployments, you can't do any of this.
- jegoodwin3 10y agoI agree testing and automation are good. I think they need to go beyond this to formal verification, for something on this scale and reliability. NASA doesn't make these sorts of mistakes. By the way - this is not just Amazon's problem now. We know the internet has a single point of failure. So does a lot of IoT. When will we experience the first Suicide DevOps?
- gmac 10y agoNASA doesn't make these sorts of mistakes https://www.wired.com/2010/11/1110mars-climate-observer-report/ https://www.wired.com/2010/11/1110mars-climate-observer-repo...
- lisper 10y agohttps://www.youtube.com/watch?v=6OalIW1yL-k https://www.youtube.com/watch?v=6OalIW1yL-k (Specifically https://www.youtube.com/watch?v=6OalIW1yL-k#t=3m https://www.youtube.com/watch?v=6OalIW1yL-k#t=3m but it's worth watching the whole clip (or even the whole movie) if you haven't seen it before. It's from Terry Gilliam's "Brazil".)
- icebraining 10y agoAlmost twenty years ago, though.
- pron 10y agoWell, they've had plenty of opportunities to learn from their mistakes; Amazon hasn't had this long.
- halomru 10y ago>We know the internet has a single point of failure. It has? I have yet to see the day where I can neither reach my email provider nor Google nor Hackernews. My local provider might screw up occasionally, or some number of of websites go unreachable for whatever reason. But I fail to come up with anything short of cutting multiple see cables that causes more than 50% of servers to be unreachable to more than 50% of users.
- jsudhams 10y agohttps://en.wikipedia.org/wiki/2008_submarine_cable_disruption https://en.wikipedia.org/wiki/2008_submarine_cable_disruptio...
- pron 10y agoAmazon do formally verify AWS (they use TLA+), which is probably why this failure is a human error. Of course, you could expand the formal analysis of the system to include all possible operator interactions, but you'll need to draw the line at some point. NASA certainly makes human errors that result in catastrophic failures. The Challenger disaster was also a result of human error to a large degree[1]; to quote Wikipedia: "The Rogers Commission found NASA's organizational culture and decision-making processes had been key contributing factors to the accident, with the agency violating its own safety rules." [1]: https://en.wikipedia.org/wiki/Space_Shuttle_Challenger_disaster#Investigation https://en.wikipedia.org/wiki/Space_Shuttle_Challenger_disas...