3 ms·
If it's too hard for the best then the concept is dead. But I don't believe it. They made some avoidable mistakes. Just look at the pattern emerging from these
by fauigerzigerk 16y ago
If it's too hard for the best then the concept is dead. But I don't believe it. They made some avoidable mistakes.
Just look at the pattern emerging from these kinds of incidents. There's an automatic cluster recovery mechanism that works for individual node failures but makes matters worse once a larger number of nodes fail.
I wonder whether they did extensive testing or simulation of that scenario. The initial root cause is probably unpredictable because there may be many, but what follows is not unpredictable.
I'm not ready to concede that because they are such an insanely smart elite group of people we just have to live with week long outages.
- ra 16y agoI think it's more of a learning experience. In the grand scheme of things, it's still day one for computing services like AWS.
- nicpottier 16y agoThe problem with all this is that you never hear about the many many failures that occur (daily, hourly at the scale they operate?) that don't cause an outage.
- scott_s 16y agoI think you're drawing the wrong conclusions from the grandparent post. It's not These guys are the best, so we will just have to live with these failures. It's These guys are smart people sailing in uncharted waters, so they're going to get a little lost now and then. The second conclusion implies that these waters will eventually become charted, so everyone will be able to avoid these problems in the future.
- fauigerzigerk 16y agoTo me, the parent as well as your comment sounds a little bit too apologetic. The waters are not totally uncharted and the outage was disproportionate. AWS isn't a research project, it's a commercial offering, so they have to take some blame. That re-mirroring storm was not completely unforeseeable. It's exactly the thing you have to consider when designing these kinds of systems.