8 ms·
Operational issue – Multiple services (UAE)
- potatoproduct 7mo agoWe are living in increasingly weirder times.
- dgxyz 7mo agoNot really. It's just been pretty damn quiet for years.
- debo_ 7mo agoWhen I first learned that there were AWS Middle East regions, my first thought was "wow they are more optimistic than I am ."
- Cyph0n 7mo agoNo, they are more aware of the customer demand for compute in the region.
- toast0 7mo agoGoogle Cloud also has middle east locations. As does Azure, Oracle and Alibaba. Afaik, IBM Cloud does not. I think those five and AWS are the top 6 global public access clouds.
- astrange 7mo agoA factory not working because of a missile strike seems pretty classic actually.
- guerrilla 7mo agoSure, but it's not a factory.
- mediaman 7mo agoWhy not? It's a physical building with lots of equipment that produces products shipped to its customers. Its products are sequences of electrons, instead of atoms. But so are power plants. And in the context of what happens when they're hit by missiles, a factory, data center, and power plant all behave the same.
- astrange 7mo agoIt's a big building with a lot of capital assets inside that are the means of production for a business…
- buttermeup 7mo agoNot really, although things may be trending in the wrong direction. Pick any “weird” current event and you can find much worse examples in relatively modern history. Just compare whatever you think of to the siege of Leningrad, for instance.
- Shank 7mo agoI wonder if this was bad targeting job or intentional. I appreciate the transparency and optimism in the status updates though!
- general1465 7mo agoIn Southern Europe some smaller web servers are intermittently not working, while big servers like YouTube are working fine. But I don't think it is related.
- Trasmatta 7mo agoIs this the one in Bahrain?
- the_mitsuhiko 7mo agoUAE. Abu Dhabi
- ejdyksen 7mo agoJust one AZ, not the whole region: > The other AZs in the region are functioning normally. Customers who were running their applications redundantly across the AZs are not impacted by this event.
- boxedemp 7mo agoAmazon usually has 3 AVs per region, looks like there are surviving AVs but the system didn't switch over gracefully. I bet that was an interesting sev2 ticket!
- easton 7mo agoIt depends on the service if things move gracefully or not. The incident explains it's only EC2 (and dependent services) in that AZ, so if they try to route traffic for services hosted on EC2 to that AZ it's not working (and customers running instances in that AZ have lost access). The other ones are not impacted. They always like to tell you to pay for more than one instance in different AZs so if this happens you don't get impacted.
- etyhhgfff 7mo agoI doubt that there is enough capacity to evict a full AV to another one.
- boxedemp 7mo agoThere's supposed to be for many (not all) use cases.
- eptcyka 7mo ago> one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. God forbid we'd ever say that it was struck by a missile or a munition in an act of war.
- hdgvhicv 7mo agoMaybe a missile, maybe a drone, maybe debris Doesn’t really matter, we know trumps latest war is the cause
- NikolaNovak 7mo agoThat's what I'm trying to understand too. It's this a meteor,tree,etc? Or a human made object,and if so accidental or intentional one. Further risk assessment would be dependent on root cause.
- rrvidqdi 7mo agoThe earliest phrasing I saw internally was "Root cause is identified as a drone attack to DXB61 site". That's somewhat open to interpretation, and could also have simply been incorrect. It was scrubbed from the ticket, though, and it now merely vaguely gestures toward a "power event". The ticket I'd expect to have further detail was locked down.
- bigyabai 7mo agoPotential moon bear attack; we're waiting on satellite imagery to confirm it. https://youtu.be/pvjgIxuVdo4?t=96 https://youtu.be/pvjgIxuVdo4?t=96
- mrguyorama 7mo agoThe actual Whitest Kids You Know Youtube channel has been reuploading old skits with the title changed and maybe a single word audio dub as direct criticisms of this administration's horseshit. Their version from 2 days ago: https://www.youtube.com/watch?v=S3o-3pL79O4 https://www.youtube.com/watch?v=S3o-3pL79O4 It's insane. Jokes written 20 years ago work better today. It's been extremely on point and it's sad how much criticism from the mid 2000s which was dismisses as "Not supporting America" was actually 100% on point and still fully valid and correct and prescient and on point 20 years later because the same exact old assholes are still in power and calling the shots and finally allowed to go weapons free on the geopolitical agenda's they've held for decades. People didn't listen that the Bush Jr admin, who started middle east wars under fabricated pretenses and booted up the surveillance regime with the PATRIOT act and called war protests "Anti-american", was Fascist and they still aren't listening as a racist, sexist, rapist moron drives the US off the deep end and starts even more illegal and false wars and directs that same DHS Bush created to destroy the enemies of the state within.
- p-o 7mo agoInteresting adjacent theory is how much are datacenters becoming military target to strike as part of disrupting initial defenses. It doesn't seem it was the case in this instance, but I could see this becoming a more important target in future. Seems like it should be somewhat easier to bomb 50 datacenters than it would be to hack and disrupt 1000s of different services. Again, this is just me thinking out loud on a tangent and this doesn't have much to do with this story, but I felt it was an interesting thought to share nonetheless.
- Imustaskforhelp 7mo ago> Seems like it should be somewhat easier to nuke 50 datacenters than it would be to hack and disrupt 1000s of different services. The bigger part of me seems that if we someone nukes 50 datacenters all at once or say all of Amazon's datacenters at once, then the data stored in there would simply be gone and given so many datacenters are located in Virginia,USA iirc or just so many companies being reliant on few datacenter providers. The larger threat to me with the lose of data is firstly the panic within public fronting services but also, with Hedge Funds, Pension funds or banking datacenters who might be using these and if they lose the data, then its gonna cause even more public mayhem. Some might be saying oh off-site backups exist but there has atleast been one instance, where a single Google accident had led to massive issues for a 135 Billion $ pension fund. Relevant Kevin Faang video about it: https://www.youtube.com/watch?v=3GOAUyipnM4 https://www.youtube.com/watch?v=3GOAUyipnM4 [Google Accidentally Deletes $135 Billion Pension Fund, Chaos Ensues]
- fc417fc802 7mo agoNotably they did have backups. As you would expect for a $135 billion dollar undertaking. It's just that restoring from a calamity tends to be time consuming (a key difference between failover and a backup).
- treesknees 7mo agoIn my experience many middle eastern companies tend to only operate out of the Middle East AZ’s. They’re not backhauling their data and customers to us-east-1. If the goal is to severely disrupt middle eastern rivals, then you don’t need to hit every possible AWS datacenter.
- anonu 7mo agoWe have business in UAE. For whatever reason I defaulted to us-west-2 since these particular applications are not latency sensitive.
- Imustaskforhelp 7mo agoHas this ever happened ever in history of Cloud providers before this because of war? They mention that the datacenter had fires and sparks and they are mentioning hours of downtime but given the situation, How does that prevent the situation from happening again. It's best for people to use safer regions than the middle east in the moment as missiles might target the same datacenter seeing that some damage was caused. Moving forward, will there be a demand (all be small) for nuclear bunker esque datacenters which can withstands missiles? I know absolutely nothing about constructing underground but can explosives not be used to create underground datacenters comparatively cheaply? One can also use revamped Nuclear bunkers (although the scale of AWS datacenters might be huge tho who knows) Had some ideas which show that this idea might be interesting, https://www.nature.com/articles/s44284-026-00406-2 https://www.nature.com/articles/s44284-026-00406-2 I am curious but what are the safety attempts made by Internet Exchange Providers or (had to search it up) but Submarine Cable landing stations, to me it feels like blowing these up leads to internet downtime across whole country / between providers.
- deleted 7mo ago[deleted]
- tbrownaw 7mo ago> but given the situation, How does that prevent the situation from happening again You don't. Instead, you make sure your failover or DR setup is regularly tested and works.
- toast0 7mo agoHistorically in the US, some portion of Bell installations were designed to be resistant to attack. But it comes at large expense for construction and maintenance. Underground facilities also bring increased risk of flooding. Competition and deregulation and lack of attacks leads towards less robust installations to reduce costs. Geographically redundant installations help as long as all installations aren't targetted; and are valuable for operational concerns other than just attacks.
- userbinator 7mo ago
- bgentry 7mo agoThe important quote from the timeline: Mar 01 9:41 AM PST We want to provide some additional information on the power issue in a single Availability Zone in the ME-CENTRAL-1 Region. At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. The fire department shut off power to the facility and generators as they worked to put out the fire. We are still awaiting permission to turn the power back on, and once we have, we will ensure we restore power and connectivity safely. It will take several hours to restore connectivity to the impacted AZ. The other AZs in the region are functioning normally.
- Imustaskforhelp 7mo ago> we will ensure we restore power and connectivity safely this would require human intervention and I am a bit worried what if the strike can happen again and human lives might be lost. IIRC there have been cases in history where sometimes a same location is targeted across multiple days. Obviously, AWS might have local employees working in the region but would there be an evaluation of this threat itself within the relevant team in AWS. What if they try to bring the service back but then missiles are struck again and what if human lives might be lost on it. Let's just hope that it could be part of a evaluation as well.
- tokyobreakfast 7mo ago> this would require human intervention that's the difference between heroes and ordinary employees who bitch about having to go into the office twice a month. same as the stories you hear of guys taking snow-cats up a mountain in a blizzard to restore phone circuits or radio transmitters gone offline.
- flymasterv 7mo agoMan, don’t be a “hero” trying to restore a lower ping to someone trying to buy a kindle in Jeddah.
- 7mo ago
- seydor 7mo agoIran may be heavily overpowered but they play their cards well. They will lose the war but the economic damage to the gulf is irreparable
- roughly 7mo agoThey've been preparing for this for 40+ years - this has been the existential threat for Iran since the revolution. You're right they can't win, but I'd expect this to get messy.
- DANmode 7mo ago> they play their cards well. They will lose the war so…
- itemize123 7mo agoso they self-destruct to force better terms of negotiation.
- DANmode 7mo agoI’ve done some light game theory reading - but this isn’t covered - if they’re gone, who are they getting better terms for?
- Imustaskforhelp 7mo ago> I’ve done some light game theory reading - but this isn’t covered - if they’re gone, who are they getting better terms for? I think the idea can be that they are getting worse terms for America. Saddam Hussein from Iraq lost as well but the war was so messy that so so many americans died in bloodshed and gruesome ways and there was a whole gulf war syndrome within Army with severe PTSD. This makes the people highly unreceptive of war and the govt should have more pressure from people. So in theory, America should've thought twice before attacking the same region. I also think, that its hard to ever really remove an ideology. You definitely see ideologies of the past spring up in future and I think that sure they might lose the govt power but they would still have some influence for what its worth. Also, Ayatollah was killed by these forces. Atleast the highly religious of people of Iran/politicians might spin the story to become religious which does have such ideas of inflicting maximum damage to the opponent at times. I don't think that game theory can be applicable for religious/personal matters at times
- deleted 7mo ago[deleted]
- deleted 7mo ago[deleted]
- DataDaemon 7mo agobut but they said this is a cloud!
- joeblubaugh 7mo agoThis has recently been updated: > Due to the ongoing conflict in the Middle East, both affected regions have experienced physical impacts to infrastructure as a result of drone strikes. So they're confirming actual aggression