3 ms·
For me-south-1 (Bahrain), all 3 data centres providing the redundancy were blown up by Iran.[1] The redundancy was localised to small geographic area and a sing
by dhx 18d ago
For me-south-1 (Bahrain), all 3 data centres providing the redundancy were blown up by Iran.[1] The redundancy was localised to small geographic area and a single government--something customers of AWS were hopefully aware of when they entrusted AWS with their data.
It's always buyer beware for any claims of availability. Engineers completing a FMECA[2] will (or should) always state upfront what type of failure modes they've deliberately excluded (such as meteor strike) or else every FMECA would be full of failure modes that have never been measured, and are not worth anyone's time worrying about. These exclusions vary by application--a time capsule, seed vault, etc are intended to outlast wars and collapses of empires. Typically a bunch of data centres aren't designed to withstand such failures.
I do think however it'd be reasonable to include the prospect of war for calculating data centre / cloud service availability. Especially in a place such as Bahrain where the country is obviously concerned enough about the prospect of war to have built very permanent and expensive air/missile defence sites. New Zealand on the other hand--maybe not so important to consider.
[1] https://news.ycombinator.com/item?id=49033240 https://news.ycombinator.com/item?id=49033240
[2] https://en.wikipedia.org/wiki/Failure_Mode,_Effects,_and_Criticality_Analysis https://en.wikipedia.org/wiki/Failure_Mode,_Effects,_and_Cri...
- 0cf8612b2e1e 17d agoAs far as I know, the attacks happened at different times. If Amazon knew that they had lost some data redundancy, shouldn’t they have been quickly mirroring that out of the region?
- testplzignore 17d agohttps://aws.amazon.com/compliance/data-privacy-faq/ https://aws.amazon.com/compliance/data-privacy-faq/ "You choose the AWS Region(s) in which your content is stored. You can replicate and back up your content in more than one AWS Region. We will not move or replicate your content outside of your chosen AWS Region(s) without your agreement."
- ericpruitt 17d agoThat would be a legal nightmare. They don't necessarily know what customers' data residency requirements are.
- bumblehean 17d agoThis. We have (well, had) customers running in me-south-1 and once the first AZ went down we wanted to proactively move their data to other regions even just as cold backups. But our legal department slapped that down pretty quickly.
- sparkling 17d agoMost likely, their own data residency terms prohibit this. It would be interesting to know if, when 2 out of 3 AZs got destroyed, customers got a heads up to move their data to a different region?
- leftbehind 17d agoWe received repeated, constant heads up to move our data by the first AZ much less second. The problem is that nobody is storing data in Bahrain unless there are data residency requirements for it. nobody wakes up one morning and chooses to launch instances, CDN or S3 and would choose Bahrain as that without a requirement to, we were contractually and legally forbidden (in the middle as a vendor) to copy even encrypted data where we don't have the key out for redundancy, so the best we could do was tell our subcustomers to download all of their buckets to their office or some employee laptops at their office
- nunez 17d agoI believe regional DR is the customer's responsibility per their Shared Responsibility Model.
- zmgsabst 17d agoAZs weren’t meant to be disaster resistant, eg, an earthquake or hurricane could take out a whole region. Regions were always the scale of disaster isolation on AWS.
- flumpcakes 17d agoRegions are really the scale of disaster isolation only in extreme cases - such as global catastrophe (meteor strike taking out a city) or in this case, when actively targeted in war. I don't really see the same thing happening to a US or European region.