5 ms·
Full disclosure: I work for an AWS competitor. While none of the specific AWS systemic failures may themselves be foreseeable, it is not true that issues of th
by bcantrill 14y ago
Full disclosure: I work for an AWS competitor.
While none of the specific AWS systemic failures may themselves be foreseeable, it is not true that issues of this nature cannot be anticipated: the architecture of their system (and in particular, their insistence on network storage for local data) allows for cascading failure modes in which single failures blossom to systemic ones. AWS is not the only entity to have made this mistake with respect to network storage in the cloud; I, too, was fooled.[1]
We have learned this lesson the hard way, many times over: local storage should be local, even in a distributed system. So while we cannot predict the specifics of the next EBS failure, we can say with absolute certainty that there will be a next failure -- and that it will be one in which the magnitude of the system failure is far greater than the initial failing component or subsystem. With respect to network storage in the cloud, the only way to win is not to play.
[1] http://joyent.com/blog/network-storage-in-the-cloud-delicious-but-deadly http://joyent.com/blog/network-storage-in-the-cloud-deliciou...
- bigiain 14y agoFWIW, once Amazon decided that an availability zone is an unreliable unit (and they tell you this up front, and strongly suggest running multiple AZ architectures for anything where you require reliability), then any cascading failure mode in a single AZ is not something you'd expect them to spend too much time protecting against. Sure, the cascade from EBS faults to RDS and ELB meant this affected more of their single AZ customers than it would have otherwise, but anyone using a singe AZ knew upfront that Amazon advised against that, and never claimed they intended to provide high-availability in single AZs. So yeah, you're right - these systemic failures can be anticipated, and Amazon's advice (to spread your important infrastructure across multiple AZs) would have protected users from most of this. (I feel quite a lot of sympathy for the engineers involved in the cross-AZ failures this incident revealed - the multi-AZ RDS and ELB failures are things customers doing everything they were told got bitten by anyway, and are probably rightly annoyed...)
- seldo 14y agoWhile Amazon do indeed say that multi-AZ is the way to go, their last 3 major incidents (including last year's cloudpocalypse) have all been full-region incidents. IMHO, their biggest design problem is that they build their systems on top of each other (e.g. ELB is built on EBS and EIP). So when one system goes down, it takes down half a dozen others -- this is especially true of EBS, and especially dangerous because the services it takes down, like ELB, are the services people are supposed to be using to route around EBS failures.
- acdha 14y agoThey've had exactly one true region-wide failure - that 17 minute routing failure earlier this year. Running a truly multi-AZ setup has avoided every other outage popularly reported as “the cloud is falling”. Some services - e.g. Heroku - have lots of impacted customers but that's due to their architecture, not the underlying AWS.
- eclipticplane 14y agoMostly, but not entirely. Under the "Impact on Amazon Relational Database Service (RDS)" section they state that due to two other bugs, some Multi-zone RDS instances did NOT fail over, because their systems knew that the masters had stopped replaying to the standbys, but had continued to process some transactions (albeit more slowly, as the failures cascaded and amplified), and thus prevented the automatic promotion of a standby, as the standbys were now out of date.
- acdha 14y agoI agree that's a problem but calling it a region-wide failure is an unhelpful exaggeration, particularly in the case of a SQL database where there's always a tradeoff between availability and consistency.
- seldo 14y agoI'm sorry, but that's just not true. Prior to last week's incident: June 28th 2012 - region-wide failure due to power outage and EBS dependency issues hitting ELB and RDS: http://aws.amazon.com/message/67457/ http://aws.amazon.com/message/67457/ March 15th 2012 - 22-minute, region-wide networking interruption (no status report available) April 2011 - cloudpocalypse, also a cascading EBS failure: http://aws.amazon.com/message/65648/ http://aws.amazon.com/message/65648/
- antirez 14y agoVery wise comment IMHO. More complex = more complex failure modes. The only way to improve is Less complex. Well also probably local storage has a number of other advantages from the point of view of performance.
- dchichkov 14y agoI don't think that there is an obvious way to make it less complex. But complexity is not inherently bad. Internet is very complex, yet it is robust. What feels wrong here, is that Amazon systems are too trusting to other Amazon internal systems.
- antirez 14y agoMoving storage locally seems like a good move. When complexity is still too high and there are no ways to make it simpler (I don't think that's the case, btw) relax your requirements.
- DanielRibeiro 14y agoI believe the damage of cascading failures was also masterfully exposed during your talk on QCon Last year: http://www.infoq.com/presentations/Debugging-Production-Systems http://www.infoq.com/presentations/Debugging-Production-Syst...
- pbhjpbhj 14y agoStart at ~10:20 for the cascading failures. Very entertaining IMO. But I would suggest the speaker cut the off-hand ostracism of large groups of people.
- digeridoo 14y agoTo be honest, EBS is the special sauce that makes me stick with AWS. There are so many little things that become so much easier by having network storage. To me the cost of upgrading or migrating machines using local storage is just not worth the perceived reliability gain to me.
- dmpk2k 14y agoIt's not just reliability though. The quality of I/O is also better. You don't see such slow I/O, with crazy fluctuations, with local storage.