8 ms·
AWS Says It Can't Restore Some Data from Mideast Facilities Struck by Iran
- markive 17d agoDoes this mean that even with 3 availability zones for Amazon S3 storage, that some data is lost?
- OrangeDelonge 17d agoDid they say its S3 data? Could also be single-az EBS or RDS.
- MiroslavPokorny 17d agoObviously what you understand is different from the reality after you actually follow all the footnotes.
- dhx 17d agoFor me-south-1 (Bahrain), all 3 data centres providing the redundancy were blown up by Iran.[1] The redundancy was localised to small geographic area and a single government--something customers of AWS were hopefully aware of when they entrusted AWS with their data. It's always buyer beware for any claims of availability. Engineers completing a FMECA[2] will (or should) always state upfront what type of failure modes they've deliberately excluded (such as meteor strike) or else every FMECA would be full of failure modes that have never been measured, and are not worth anyone's time worrying about. These exclusions vary by application--a time capsule, seed vault, etc are intended to outlast wars and collapses of empires. Typically a bunch of data centres aren't designed to withstand such failures. I do think however it'd be reasonable to include the prospect of war for calculating data centre / cloud service availability. Especially in a place such as Bahrain where the country is obviously concerned enough about the prospect of war to have built very permanent and expensive air/missile defence sites. New Zealand on the other hand--maybe not so important to consider. [1] https://news.ycombinator.com/item?id=49033240 https://news.ycombinator.com/item?id=49033240 [2] https://en.wikipedia.org/wiki/Failure_Mode,_Effects,_and_Criticality_Analysis https://en.wikipedia.org/wiki/Failure_Mode,_Effects,_and_Cri...
- 0cf8612b2e1e 16d agoAs far as I know, the attacks happened at different times. If Amazon knew that they had lost some data redundancy, shouldn’t they have been quickly mirroring that out of the region?
- testplzignore 16d agohttps://aws.amazon.com/compliance/data-privacy-faq/ https://aws.amazon.com/compliance/data-privacy-faq/ "You choose the AWS Region(s) in which your content is stored. You can replicate and back up your content in more than one AWS Region. We will not move or replicate your content outside of your chosen AWS Region(s) without your agreement."
- ericpruitt 16d agoThat would be a legal nightmare. They don't necessarily know what customers' data residency requirements are.
- bumblehean 16d agoThis. We have (well, had) customers running in me-south-1 and once the first AZ went down we wanted to proactively move their data to other regions even just as cold backups. But our legal department slapped that down pretty quickly.
- sparkling 16d agoMost likely, their own data residency terms prohibit this. It would be interesting to know if, when 2 out of 3 AZs got destroyed, customers got a heads up to move their data to a different region?
- leftbehind 16d agoWe received repeated, constant heads up to move our data by the first AZ much less second. The problem is that nobody is storing data in Bahrain unless there are data residency requirements for it. nobody wakes up one morning and chooses to launch instances, CDN or S3 and would choose Bahrain as that without a requirement to, we were contractually and legally forbidden (in the middle as a vendor) to copy even encrypted data where we don't have the key out for redundancy, so the best we could do was tell our subcustomers to download all of their buckets to their office or some employee laptops at their office
- gregw2 16d agoI think so. Although the more paranoid AWS customers who turned on (and pay for) S3 cross region replication or similar cross region DR for other services would be fine.
- curuinor 17d agoThey guaranteed 11 9's durability, didn't they? e: Yep https://aws.amazon.com/s3/storage-classes/ https://aws.amazon.com/s3/storage-classes/
- deleted 17d ago[deleted]
- jonahx 16d agoI don't think this has any teeth. They don't compensate in the event of loss afaict.
- rbanffy 16d agoConsidering all the data they have globally, they might still be compliant.
- stackskipton 16d agoEven if they have payable SLA on this, most SLAs have Acts of God and Acts of War exemption.
- lbreakjai 16d agoBut do they have Act of Special Operation exemptions?
- eastbound 16d agoI'm going to reword my Terms of Service this second to add "any military operation" next to "acts of war". But I'm sure we'll then have to demonstrate whether paramilitary are assimilated to the military.
- MisterMunchkin 16d agoI was just reading an insurance policy and it said "any war, including undeclared wars" The insurers always know how to weasel out of it.
- HDBaseT 17d ago[dead]
- chews 17d agoI'm sorry but your data is in another castle.
- Chance-Device 16d agoThis should have been in that super Dario game.
- carefree-bob 16d agoThis is the flipside of data residency requirements that countries are now starting to require. If the EU wants to keep data in the EU, then great, but when the war comes and energy and infrastructure are hit, people would have wished for backups in North America, Asia, and the middle east.
- kibwen 16d agoFor long-term backups you want offline cold storage in an underground facility in a friendly jurisdiction, not a datacenter.
- bigiain 16d agoWe've beer-o-clock wargamed this a bit. If I had an "important enough" client, I think I'd store all out local (Sydney + Melbourne AWS cross region) data to AWS Singapore (to protect against Australian jurisdictional and political risks) and to a non AWS cloud provider in the EU somewhere. I reckon thatd be close to as resilient a pile of hard drives in an underground bunker, for significantly less setup and ongoing cost, while also being much more available when needed. (Can you imagine the queue at the underground bunker when multiple AWS regions get bombed? Or even imagine getting to the bunker in "a friendly jurisdiction" while a shooting war is taking place?) We haven't worked out a decent solution to Visa and Mastercard payment network going down for more than a couple of cloud billing cycles though.
- mitxela 16d agoMore likely than the network going down is you getting banned from the network because someone thought you were selling porn.
- watwut 16d agoWar did not randomly came. America intentionally caused it. And has lawless goverment and unaccountable tech industry making it bad place for data.
- 16d ago
- wewewedxfgdf 16d ago[flagged]
- knorker 16d agoWhat's you point? That if you had run your own DC in that region (because that was your business requirement) then you'd have better missile defense than AWS? Or maybe AWS or DIY, you are always responsible for geographic diversity? Anyone losing data over this lost it because they'd literally told AWS to only store it in one place.
- shevy-java 16d agoHow about not bombing other countries and then acting surprised when retaliation happens? I mean clearly the problem isn't AWS as such - it is the problem that someone leading a country is totally clueless about the world. Only personal profit is in the interest of the orange clown.
- culi 16d ago> I mean clearly the problem isn't AWS as such I get your main point, but just wanna point out that AWS is one of the largest military contractors in the world. They hold multi-billion dollar contracts from the DoD, USAF, CIA, and more. An estimated $4B a year in military spending goes to AWS
- wewewedxfgdf 16d agoThe point is that AWS has the same problem as Wildberries. There is no difference at all between Wildberries and AWS data centers. If you don't know what Wildberries is then go watch their facilities systematically destroyed on YouTube - centralisation is a target. If your organization runs on AWS then you should have a contingency plan for the data center being destroyed by drones. Is that on your risk management plan?
- knorker 16d ago> If your organization runs on AWS then you should have a contingency plan for the data center being destroyed by drones. If your organization runs on servers in your basement you should have a contingency plan for flooding, fire, copper thieves, diesel shortages, etc… etc… etc… Or are you basically saying that the only safe place is outsourcing your ops to a mid sized operator? Too small and nobody will pay for the every day risks. Too big and it's a war target? Ok, let's continue that plan. Now military users move their workloads to the mid sized operators for the exact same reason you did. Oh no, we're back at square 1. A plan of putting all your eggs in one basket is never good. And it's completely orthogonal to AWS vs the non-AWS options. Pretty basic stuff. > If you don't know what Wildberries is Not exactly esoteric knowledge. But it's also not the same thing. Wildberries could not "back up" their inventory to an offsite location with the footprint of a suitcase. > If your organization runs on AWS then you should have a contingency plan for the data center being destroyed by drones. Is that on your risk management plan? Sure, if you discard ALL other risks, that happen every day, leaving only war as the remaining risk to manage, then your risk management plan makes sense. But the other risks are still there. Your DC operator going bankrupt and having their power cut is a risk that didn't go away.
- Art9681 16d agoNo disaster recovery plan? No offsite backups? Someone failed to applied the most basic principles that have existed for decades.
- rbanffy 16d agoThe more dramatic contingency you have to plan for, the more expensive the plan gets. Earlier this week I mentioned that if we lose enough data centres to bring our operation down, the first items in the to-do list becomes securing weapons, vehicles and fuel.
- ares623 16d ago"Daddy, where were you when the flames reached our house?" "I was in the office, reviewing Terraform plans"
- jiggawatts 16d agoI had the same discussion with a manager about the backups of financial contracts for cleaning school facilities. He just couldn't get past the notion that if the six copies in four buildings across two states were all simultaneously physically destroyed, then most likely there are also no more schools left standing, and hence the contracts to clean them are null and void. Also, payment is now in booze and ammunition, not dollars.
- chasd00 16d ago> to-do list becomes securing weapons, vehicles and fuel. I toured a datacenter once back in the early 2000s and they showed me 30 days of generator fuel storage. When i asked them why 30 days and not 35 they replied "we're such a major customer of both electricity and fuel that if we don't get electricity or fuel for 30 days there's way bigger problems than your website not being online" hah.
- mr_mitm 16d agoThat's probably already true for 7 days or less
- michael-bey 16d agoWhat a nightmare scenario to tabletop. How do you even begin to recover from something like this?
- NegativeLatency 16d agoBackups in a different region?
- noir_lord 16d agoWorks unless local laws specifically block you doing that which they do for some classes of data in some countries. Multi-cloud in the same country (if that exists in the country and is far enough apart) maybe.
- criemen 16d agoDo cloud providers even share data center locations so you can assess the "far enough" bit yourself?
- flumpcakes 16d agoNo - and usually the reason is so they cannot be targeted.
- toast0 16d agoYou usually get city level location information. Depends on your definition for 'far enough' if that works for you. me-south-1 is about 250 miles away from me-central-1, but that's not far enough in this instance. Given that, I think city level location information should be good enough. 250 miles is pretty good for weather or not specifically targeted destruction (wildfire / industrial explosions / arson), but it's clearly not enough if your data is in a building targeted in a regional war. Assuming datacenters remain targets in wartime, I think it's fair to assume if one datacenter in any particular country is attacked, all the rest of the datacenters in that country are likely to be attacked, too. In that case, offline storage (tapes and things) in inconspicuous locations might be the way.
- rbanffy 16d agoWell… they have a good excuse.
- vdfs 16d agoAnd witness
- shevy-java 16d ago[flagged]
- Cyclone_ 16d ago[flagged]
- mitxela 16d agoIsrael had been trying to get every president to bomb Iran for decades. There's a reason they wouldn't do it themselves. They finally got a president stupid enough to listen.
- drnick1 16d ago[flagged]
- carefree-bob 16d agoThis is not exactly a nuanced view of the conflict, and in either case, the fact that you don't like that someone on the other side of the world is chanting death to America doesn't give you a bonus card for a free attack. Seriously, it's like people, when deciding whether to launch a war or not, are not thinking "how will the other side react and will this conflict benefit me" but instead they are only thinking "does this nation deserve to get hit". Well, news flash, your moral outrage does not translate into you not suffering more than your opponent during a conflict. It's a completely separate issue, and a personal issue between you and your priest or rabbi. When it comes to starting wars, you have to look at military capabilities and long term outcomes, not "does this nation deserve to be attacked".
- deleted 16d ago[deleted]
- _hyn3 16d agoThis is not exactly a nuanced view either and claiming that someone needs to discuss a personal issue with their clergy doesn't reduce the temperature or elevate the discourse. This was not "a free attack". The goal was and is preventing the world's leading terror organization from acquiring nuclear weapons, especially when they already have the missiles to carry them. It's just a bad situation and the decisions are difficult. Even now, the IRGC continues attacking their erstwhile allies. Those would likely be nukes if they'd had them.
- rvz 16d agoBackup both locally and everywhere no matter what.
- burnt-resistor 16d ago50%+ of companies that lose all of their data go out of business in 6 months. DR/BCP costs are readily justified by doing a Business Impact Analysis (BIA).. budget up to some fraction of risk cost * risk probability. And a friendly reminder that replication isn't a tested data backup.
- rishikeshs 16d agoI think this is due to the data residency requirements in UAE. I'm working with a client in the health space and the government requirements requires me to store data only in UAE! Tried with AWS but they were not allowing any new instances and I had to go with Azure.
- jackb4040 16d agoHi, I'm from the future. You might want to consider storing the data somewhere besides an Azure datacenter in the UAE.
- r_lee 16d ago> the government requirements requires me to store data only in UAE!
- birdatlaw 16d agoturns out reading comprehension skills have still not gotten better in the future
- jackb4040 16d agoIn the future, some companies begin to store their data on-premises away from big centralized datacenters. But many companies do not, due to costs and the general friction of changing how things are done. If OP tells me the name of his company I can hop in my time machine and tell him how it plays out.
- noeltock 16d agoMENA is the on-prem capital of the world, don't worry.
- instakill 16d agooff-topic but nice username. Fan of Charlie Kelly?
- SmirkingRevenge 16d agoIt was always a bad bet for billionaires like Bezos to become Trump enablers. You weren't buying a seat at the table, or the privilege of being left alone, you were just signing yourself up to be force-fed shit sandwiches over and over (And the shit-to-bread ratio gets worse as time goes on) You should have used your considerable resources to fight. If only billionaires would oppose aspiring tyrants with the same zeal with which they oppose even minor tax increases.
- CamperBob2 16d agoHe had little choice. The Trump tariffs could've been a massive, massive blow to Amazon, so I'm sure he felt he had to get out in front of them and buy some influence with the incoming administration. See also Tim Cook. Doesn't make it right to suck up to Trump, but it was, and unfortunately still is, a rational move.
- mitxela 16d agoBezos hasn't lost anything from this. He's only gotten richer.
- SmirkingRevenge 16d agoWell, he did lose some data centers. Those aren't cheap. The sort of global instability that is being created by all this is bad for everyone. It generally isn't good for business either. We're so unstable and capricious now, countries are actually trying to decouple from American tech services like AWS and Amazon. That ain't great for Bezos. Maybe inertia kicks in after Trump and things revert to the mean, but restoring confidence in the US again as a friendly, stable nation is going to be tough if we're always 4 years away from another Trump-type figure running the show.
- skybrian 16d agoI wonder if they'll start adding an underground bunker to new data centers so you can put an S3 replica there?
- dhx 16d agoHow long would it then take to be able to use the backed up data? Wait for a war to end and a replacement data centre to be built...? By that time most data probably no longer matters (e.g. business no longer exists). It's more likely the entire data centre (not just backups) would need to be built underground (or cut and cover) at enormous expense. A price that perhaps for certain data sovereignty reasons the government of Bahrain (or companies in Bahrain requiring it) would be happy to pay? Another way to do things on the cheap could be small-scale "covert hosting". Buy an apartment or house, maintain it to give an outside appearance of being an apartment or house, but inside it has a few racks of IT equipment. This has been done in the past for telephone exchanges in some countries, not for security reasons, but rather to hide an ugly bit of infrastructure that due to technology limitations of the time had to be located deep within a residential neighbourhood.
- cmiles8 16d agoThis interview with an AWS leader isn’t aging well, from CBS Sunday morning: Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!" https://www.cbsnews.com/news/cloud-computing-loudoun-county-virginia/ https://www.cbsnews.com/news/cloud-computing-loudoun-county-...
- Betelbuddy 16d agoClearly since the MBAs took over AWS standards are not anymore what they used to be. That marketing guy should not be talking to the press, as he does not have the skills, and if somebody happens to say...our data center we wont lose any data if we have an issue, without qualifying it will depend on what quality of service, and usage of our services you setup ...is the type of technical answer that should make a hiring interview stop at the moment. He is also violating an enormous amount of compliance requirements, by disclosing the location of the data center, and having strange people inside making a tour. Did he vet the crew and their accompanying party? Did one of them accidentally left some kind of device within the insider perimeter? There at least one or two ISO certifications he is violating there. As customer I would be asking questions... AWS always made very clear they wont copy your data to another region as only you know what your compliance and data residency requirements are. But at the same time they always said, its up to you to come your with your disaster recovery strategy based on your project requirements. And it has always been the case copying your critical data to another region is one of the first things on your check list. And their Well Architected Framework and other docs make this plenty clear: "It is a good practice to always make backups of your data, and copy these to another site (such as another AWS Region)." Also... "All DR strategies require that data sources are backed up within the AWS Region, and then those backups are copied to the recovery Region." And also for single-Region / Multi-AZ architectures: "Where possible, you should also copy data backups to another AWS Region as an additional layer of protection." "AWS Architecture Blog — Disaster Recovery Architecture on AWS, Part II" has a whole section named "Backup to another AWS Region": "By copying your data to another Region, you can handle the largest scope of disasters." https://aws.amazon.com/blogs/architecture/disaster-recovery-dr-architecture-on-aws-part-ii-backup-and-restore-with-rapid-recovery/ https://aws.amazon.com/blogs/architecture/disaster-recovery-... Or "Creating backup copies across AWS Regions" - https://docs.aws.amazon.com/aws-backup/latest/devguide/cross-region-backup.html https://docs.aws.amazon.com/aws-backup/latest/devguide/cross... This whole thread of people literally saying , "on no I trusted them...I did not know they could lose my data", with no technical context...is the the kind of incompetence I would expect from a generation raised on vibe coding and llm prompt driven miseducation...
- chasd00 16d agoThey say "some" data, i wonder what percentage that really is. I haven't seen pictures but I find it hard to imagine all of me-south-1 was completely leveled to the point where's there's just nothing left. On the other hand, if you have 100 rows of racks and then randomly take out a contiguous 10% across both rows and columns it may be functionally equivalent to taking out everything.
- tgsovlerkhgsel 16d agoI wouldn't be surprised if the engineers said "we can probably recover between 20-30% of the data but it will cost 200 hours of engineering and the data will be 7 months old by then" and the beancounters said "we'd rather have one news cycle rather than the news watching what we can and cannot recover + save those 200 hours, we'll just say it's all gone".
- phendrenad2 16d agoUh. Uh-oh. It's not clear from their messaging if multiple availability zones were severely damaged, or if the damage to one availability zone was simply more than they planned for. If it's the latter, that's a big uh-oh. The wording certainly seems very careful: "The damage to our infrastructure spanned multiple availability zones and exceeded what our regional and multi-AZ services are designed to withstand"
- ernsheong 16d agoHey but that's exactly as per design. It is the customer's responsibility to store stuff elsewhere as DR backup, not AWS.
- mitxela 16d agoThat's the excuse they'll say, yes, then we quote back to them "eleven nines" and they come up with an excuse for that too
- tornado134 16d ago[dead]
- Kvarnek 16d agoGuess those multi-AZ promises have an asterisk when actual missiles are involved. Makes you double-check your own off-site backups.
- vanjajaja1 16d ago"..even if something happens only once in a billion requests, that means it happens multiple times per day within S3." but one in a trillion...
- crate_88 16d ago[dead]
- weinzierl 16d agome (Middle East) ├── me-south-1 (Bahrain) DOWN since 2026-04 │ ├── mes1-az1 me (Middle East) ├── me-south-1 (Bahrain) DOWN since 2026-04 │ ├── mes1-az1 DOWN │ │ └── mes1-mct1-az1 (Oman, Muscat) │ ├── mes1-az2 DOWN since 2026-03-01 │ └── mes1-az3 DOWN ├── me-central-1 (United Arab Emirates) │ ├── mec1-az1 │ ├── mec1-az2 DOWN since 2026-03-01 │ └── mec1-az3 DOWN since 2026-03-01 └── il-central-1 (Israel, Tel Aviv) ├── ilc1-az1 ├── ilc1-az2 └── ilc1-az3 DOWN │ │ └── mes1-mct1-az1 (Oman, Muscat) │ ├── mes1-az2 DOWN since 2026-03-01 │ └── mes1-az3 DOWN ├── me-central-1 (United Arab Emirates) │ ├── mec1-az1 │ ├── mec1-az2 DOWN since 2026-03-01 │ └── mec1-az3 DOWN since 2026-03-01 └── il-central-1 (Israel, Tel Aviv) ├── ilc1-az1 ├── ilc1-az2 └── ilc1-az3
- nerdile 16d agoOk claude, now format that in a way humans can read
- weinzierl 16d agoSorry, I accidentally posted it twice while fixing it and on a slow connection.
- dhanudhakne 16d agoYas
- dhanudhakne 16d agoIii
- dhanudhakne 16d agoOkk
- weinzierl 16d agoThe data center layout should look something like this: me (Middle East) ├── me-south-1 (Bahrain) DOWN since 2026-04 │ ├── mes1-az1 DOWN │ │ └── mes1-mct1-az1 (Oman, Muscat) ??? │ ├── mes1-az2 DOWN since 2026-03-01 │ └── mes1-az3 DOWN ├── me-central-1 (United Arab Emirates) │ ├── mec1-az1 │ ├── mec1-az2 DOWN since 2026-03-01 │ └── mec1-az3 DOWN since 2026-03-01 └── il-central-1 (Israel, Tel Aviv) ├── ilc1-az1 ├── ilc1-az2 └── ilc1-az3 Not sure about the Muscat local zone, whole me-south-1 region has been reported down despite Muscat still being operational. If someone had told me a year ago that a whole AWS region could go down I'd called them crazy, but now me is close to exactly that happening. See also previous discussion: https://news.ycombinator.com/item?id=49033240 https://news.ycombinator.com/item?id=49033240
- spbaar 16d agous-east-1 is finally looking pretty stable for once.
- gilbetron 16d ago11.3 Force Majeure. Except for payment obligations, neither party nor any of their affiliates will be liable for any delay or failure to perform any obligation under this Agreement where the delay or failure results from any cause beyond its reasonable control, including acts of God, labor disputes or other industrial disturbances, electrical or power outages, utilities or other telecommunications failures, earthquake, storms or other elements of nature, blockages, embargoes, riots, acts or orders of government, acts of terrorism, or war.
- 00deadbeef 16d agoIt's not terrorism because it's self-defence and it's not war because Donald Trump says so
- mitxela 16d agoIt's beyond the reasonable control of Jeff Bezos. Even though he supports trump with billions of dollars, he didn't ask for this particular war.
- rcbdev 16d agoI just gave the crazy man a lot of money knowing he wants to start a holy war. I did not want him to fund this particular holy war that turned out to be disadvantageous to me!
- mitxela 16d agoKind of like building your house on a floodplain and claiming insurance for a flood. They'd still say it was an act of god.
- TeMPOraL 16d agoWell, flood is an act of God. "Floodplains" is another lie invented by those "scientists" to get you pissed, probably on a break from their usual attempts to convince you magnets are not a genuine miracle.
- Eastmill 16d agoSeems like their multi-AZ redundancy didn't account for missile strikes. Good reminder to always have your own backups.
- brightball 16d agoThere are many regulations over there which prevent data from leaving the country. Sometimes those rules can have serious consequences.
- simoncion 16d agoSince roughly zero of the articles I've seen link back to the source of their quotes, here it is: <https://status.aws.amazon.com/#multipleservices-me-south-1_1789467748 https://status.aws.amazon.com/#multipleservices-me-south-1_1...>
- carabiner 16d agoEverything not saved will be lost.
- xyst 16d agoWe collectively gave this company trillions over the years and they still can’t get it right.
- deleted 16d ago[deleted]
- DeepYogurt 16d agoI wonder if this will cause a mini cyber insurance crisis. I don't think any of those data loss plans have been tested at scale.
- pembrook 16d agoAhhh the downside of “data sovereignty” rules and the de-globalization meme strikes again. I’d bet a bajillion dollars the reason this happened is due to government thinking it’s a good idea to make it illegal to store data outside their country. Hey EU, take note of this next time you create silly data residency requirements that don’t allow data to travel outside your region. Encryption is an easy solution to multi-region residency…as long as the European Commission doesn’t stupidly keep trying to make encryption illegal too! Hint: Russia absolutely knows where your data centers are.
- WA 16d agoThe EU is big enough to have several geographically independent data centers. without leaving the sovereign territory.
- pembrook 16d agoYes there’s never been division or wars in Europe and the EU will always exist… Germany and France in a debt spiral and turning inward/hyper-nationalist while massively re-militarizing means the EU is a safe place to structure your data with zero redundancy. I wouldn’t bother worrying about key industries and functions, since, as history has shown, the EU is a bulletproof institution that no country has ever left. And of course every data center in Europe has Israel-grade air defenses, it’s not like they are just sitting ducks for a fleet of drones to take out within 24 hours.
- pjc50 16d agoThis is a valid point if you're not so confrontational about it.
- pembrook 16d agoThe only valid tone to take with people who are on a crusade to ban encryption is disdain and mockery.
- rsynnott 16d agoThe EU is quite big; there are seven AWS regions there, each with three AZs.
- lexicality 16d agoThey can recover most of it? I assumed that those AZs had been offline for so long because they were like gone gone.
- purpleidea 16d agoMy personal bet is there's an 80% chance this is caused by some internal bootstrapping problem that they've messed up. AIUI all the main cloud vendors are in trouble here. The automation project I work on is expressly designed to help folks solve this DR/bootstrapping problem. Soo many people get this wrong. Of course missiles don't help things, but I'd bet AWS is primarily to blame here. I'd love an actual technical report of why they can't recover things.
- jeffrallen 16d agoI work on the same kind of thing, and while we think hard about bootstrap problems, we always find new surprising ones. The problem is you never know until you do it, and creating a faithful test of restarting giant systems is economically impossible. Because if you say to the boss, "look, I need 1 million now to test against a maybe 100 million loss, maybe in 10 years" they don't give you the money (and rightly so).
- AdamN 16d agoEven if you did the $1MM test there is very low likelihood that the $100MM event would be fully mitigated 10 years down the line (after who knows how many changes - physical, logical, and even in the org chart). The only way to approach readiness here is repeated investment - like one team doing the deep dive and another pulling cables and then constantly doing pre- and post-mortems.
- genxy 16d agoSo many AWS simps in this thread that don't actually understand how AWS scrimped out and fucked their customers. You can't all suck up to AWS at the same time, you need to serialize.
- KingOfCoders 16d agoIf all your backups are with AWS you don't have backups at all.
- expedition32 16d agoI suppose this is why Microsoft and Google want to build data centers in the Netherlands. Its not cheap but it is safe.
- pknerd 16d agoI wonder what kind of "some data" is: military installations in Arab states and Israel?
- m4rtink 16d agoStuff hosted in overpriced cloud has actually no backups ? Unbelievable!
- bojangleslover 16d agoI think people are overthinking this. I'm the first one to criticize AWS (I run a competitor, carolinacloud.io) but I don't think we should hold it against them when they lost data due to their hard drives being physically bombed.
- paulddraper 16d agoOf course not. The surprising thing is that multiple availability zones were bombed simultaneously. And to my knowledge, they haven’t even yet said 2+ AZs were compromised.
- TeMPOraL 16d agoTurns out that "availability zones" in cloud parlance seem to have nothing to do with geographical availability. Apparently they're logical splits and are more about billing than anything.
- torginus 16d agoHow often does an AZ go down while the other 2 still work?
- paulddraper 16d ago> An Availability Zone is one or more discrete data centers with separate and redundant power infrastructure, networking, and connectivity in an AWS Region. Availability Zones in a Region are meaningfully distant from each other, up to 60 miles (~100 km) to prevent correlated failures, but close enough to use synchronous replication with single-digit millisecond latency. > https://docs.aws.amazon.com/whitepapers/latest/aws-fault-isolation-boundaries/availability-zones.html https://docs.aws.amazon.com/whitepapers/latest/aws-fault-iso... They are far enough apart that tornados, floods, and fires cannot simultaneously impact multiple.
- Game_Ender 16d ago10s of miles is close enough that they are functionally in the same place in a military context though. The latest cheap massive wave attack drones fly hundreds of miles and hour which puts the DCs less than a minute apart by flight time. This means if you want protection against those risks you have to use multiple regions instead of, or in addition to multiple AZs.
- paulddraper 16d agoWhat AZs were affected? I don’t think I’ve seen them say?
- alexpotato 16d agoWhen doing Disaster Recovery (DR) planning as a SRE in the New York City area, I sometimes use the phrase "Hurricane Sandy 2" to describe an event so big that it knocks out the power/compute/etc for an entire region. I may just start using AWS Bahrain 2 as an additional example.
- cranberryjoe 16d agoPretty soon we won’t need backups at all, just have AI regenerate all the data.
- harshaw 16d agoEx AWS. AWS had one zone specific products. If those were in the affected AZ that data is lost. AWS has multiple multi-AZ products. S3 and EFS are good examples. If you loose an AZ for an extended periodic of time, the dataplane of those services will migrate shards around until you get back to your durability goals.
- at1as 15d agoS3 has 11 9s of durability (excluding the geopolitical incidents)