14 ms·
AWS Frankfurt incident
- simzor 5y agoThis reminds me that we should get ChaosMonkey up and running. :D
- slater 5y agois it on fire?
- hallway_monitor 5y agoRight? What temperatures would servers continue to operate at, but would be unsafe for humans?
- uniformlyrandom 5y agoThe hardware is heat-resistant, so probably around ~75C (167F).
- offmycloud 5y agoIf they did have a fire, they're probably waiting for all the smoke and FM200 (or other extinguishing agent) to clear out before they send anyone in.
- JacobDotVI 5y agoit might not be the fire itself, but the fire suppression system might uses gases fatal to humans and so they must wait for those gases to vent / dissipate.
- lmilcin 5y agoI was in a server room that overheated and servers started to shut down. AC failed and there was no environmental monitoring. The air was too hot to breathe. I had to take and hold breath to jump in to shut down most important assets. There was redundant air conditioning except for exactly one element: air inlet on the roof. A thin plastic bag blocked it.
- geoduck14 5y agoBRB. Going to go update my FMEA
- ta988 5y agoWas there some work after to add other inlets?
- jacquesm 5y agoClassic. $0.05 bit of junk causing a very large multiple of damage. Nice case to prove that plastic waste is hazardous though!
- lmilcin 5y agoYes, that plastic bag cost couple million. But the offset was definitely increased understanding of the concept of single point of failure.
- rdines 5y agooh my. that reminds me of this story that I hadn't thought of in a long time, LOL: https://www.datacenterknowledge.com/archives/2012/07/09/outages-surviving-electric-squirrels-ups-failures https://www.datacenterknowledge.com/archives/2012/07/09/outa...
- jacquesm 5y agoThat article references the explosion at 'The Planet' in 2008, I got caught up in that, that wasn't fun at all. Fortunately we had good off-site backups. Never knew about the squirrel angle, thank you!
- tyingq 5y agoPulling a spec sheet for a typical Dell 1U server, it has an operating range of 5°C to 40°C, and a non-operating (in storage) range of -40°C to 65°C. I imagine it would survive much higher temps, but it doesn't seem recommended. I would be wary of how reliable anything in that room might be later. Perhaps not the PCB itself, but there's typically lots of thin wire around with thin insulation.
- deleted 5y ago[deleted]
- redis_mlc 5y agoIf you can breathe, it should be ok. A lot of DCs after 2000 went from cold to warm based on the failure rates not changing. If you get a chance to tour HE Fremont (the old Apple factory), it's not cold. Factoid: most startups in SV started with a rack in a closet, that one day lost air circulation or cooling and went to the boiling point. I recommend people monitor the temp and cut 2 inches off the bottom and top of their closet door to allow passive cooling while you drive back to the office.
- ta988 5y agoI've had servers running in a 35°C room for almost a day. Curiously 3 years later only one hard drive died. And they are >6yo machines. Recent machines are supposed to handle 35°C air intake according to Anandtech and special servers can take more https://www.anandtech.com/show/7723/free-cooling-the-server-side-of-the-story/3 https://www.anandtech.com/show/7723/free-cooling-the-server-....
- uniformlyrandom 5y agoLast update: The environmental conditions within the affected Availability Zone have now returned to normal levels. We will provide further details on the root cause in a subsequent posts, but *can confirm that there was no fire* within the facility.
- samizdis 5y agoAn article in The Register [1] says it wasn't a fire, but speculates: While we lack any evidence on which to base an assertion, The Register has reported on erupting UPSes and tiny puffs of smoke leading to hypoxic gas being released into data centres. The whole point of hypoxic gas release into data centres is to deprive fires of oxygen. And as humans need oxygen, it can be a while before engineers can return to a data centre. The Register mentions this as it fits the facts offered in this incident, and with AWS’s language about “environmental conditions” preventing entry. The Register says that it'll update its article if it gets further info. [1] https://www.theregister.com/2021/06/11/aws_eu_central_1_incident/ https://www.theregister.com/2021/06/11/aws_eu_central_1_inci...
- uniformlyrandom 5y agoSounds like a status update from Fukushima.
- idownvoted 5y agoWell then Germany, you know what to do: Ban all AWS data centers!
- GauntletWizard 5y agoIf you have data in Frankfurt, now is the time to test your backups. There's going to be a massive rash of failures in the next few months as hardware that was compromised but limping along dies off.
- tyingq 5y agoBig sale on Frankfurt spot instances coming soon!
- a3n 5y agoA fire sale.
- perihelions 5y agoThat's exactly the definition. https://www.merriam-webster.com/dictionary/fire-sale https://www.merriam-webster.com/dictionary/fire-sale (": a sale of merchandise damaged in a fire")
- buremba 5y agowhy?
- notatoad 5y agothe two most likely scenarios where "environmental conditions" prevent access to a DC are fire or cooling failure. aws says it's not a fire, so probably the temperature rose above their allowable safety threshold for access. that's not good for computers either
- derefr 5y agoIt could have just been that someone pulled a fire alarm when there wasn't a fire, and then they had to wait for the Halon to disperse.
- deleted 5y ago[deleted]
- rsync 5y ago5:19 PM PDT We have restored network connectivity within the affected Availability Zone in the EU-CENTRAL-1 Region. The vast majority of affected EC2 instances have now fully recovered but we’re continuing to work through some EBS volumes that continue to experience degraded performance. The environmental conditions within the affected Availability Zone have now returned to normal levels. We will provide further details on the root cause in a subsequent posts, but can confirm that there was no fire within the facility.
- intsunny 5y agoOmg, please Amazon, switch to UTC for timestamps. It is 3AM in Germany, and I'm tired and I don't want to know what PDT is.
- plasma 5y agoI'm surprised these sites don't show a localised time with UTC next to it, even the popular statuspage.io doesn't do this, blows my mind.
- JshWright 5y agoThe other day Zoom posted a status update that listed the time in both CEST, and PST... How did they get daylight saving time right for one and not the other? PST isn't a thing for another few months yet...
- schoen 5y agoMaybe a European thought that Americans also use "S" for "Summer" rather than "Standard"?
- ksec 5y agoI am sorry I laughed so hard. Frankfurt in PDT.
- a012 5y agoAnd Google too. It makes me wonder who is their audience when they only use USA timezone while their customers are worldwide.
- plasma 5y agoI'm curious to hear if anyone's multi-az setup (RDS, ECS, etc) handled this event without much of an issue? I assume so but would be nice to know its working as expected!
- adrianpike 5y agoA handful of AZ-less managed AWS services were having a bad time(tm) during the peak of the incident, so if anyone was reliant on those they were having a bad time as well.
- clutchdude 5y agoIt affected services we have that are not located in the impacted AZ.
- gkop 5y agoLikewise, our RDS replica experienced degenerate lag, when neither the primary nor replica were in the scorched AZ.
- plasma 5y agoThat's really interesting. I wonder whether its because a large load of capacity needs suddenly in unaffected AZs put abnormal stress on things like networking to the whole unaffected AZs. My only experience there is in Azure, when they deployed patches for the Heartbleed etc issues, for a few weeks things were much slower in CPU power (our response times just shot up 20% for no reason then recovered a few weeks later) and there were network related timeouts that were abnormal and it all settled down eventually.
- xmodem 5y agoHeartbleed? I assume you mean Spectre and Meltdown
- celticninja 5y ago
- tyingq 5y ago6:54 PM PDT Starting at 1:18 PM PDT we experienced connectivity issues to some EC2 instances, increased API errors rates, and degraded performance for some EBS volumes within a single Availability Zone in the EU-CENTRAL-1 Region. At 4:26 PM PDT, network connectivity was restored and the majority of affected instances and EBS volumes began to recover. At 4:33 PM PDT, increased API error rates and latencies had also returned to normal levels. The issue has been resolved and the service is operating normally. The root cause of this issue was a failure of a control system which disabled multiple air handlers in the affected Availability Zone. These air handlers move cool air to the servers and equipment, and when they were disabled, ambient temperatures began to rise. Servers and networking equipment in the affected Availability Zone began to power-off when unsafe temperatures were reached. Unfortunately, because this issue impacted several redundant network switches, a larger number of EC2 instances in this single Availability Zone lost network connectivity. While our operators would normally had been able to restore cooling before impact, a fire suppression system activated inside a section of the affected Availability Zone. When this system activates, the data center is evacuated and sealed, and a chemical is dispersed to remove oxygen from the air to extinguish any fire. In order to recover the impacted instances and network equipment, we needed to wait until the fire department was able to inspect the facility. After the fire department determined that there was no fire in the data center and it was safe to return, the building needed to be re-oxygenated before it was safe for engineers to enter the facility and restore the affected networking gear and servers. The fire suppression system that activated remains disabled. This system is designed to require smoke to activate and should not have discharged. This system will remain inactive until we are able to determine what triggered it improperly. In the meantime, alternate fire suppression measures are being used to protect the data center. Once cooling was restored and the servers and network equipment was re-powered, affected instances recovered quickly. A very small number of remaining instances and volumes that were adversely affected by the increased ambient temperatures and loss of power remain unresolved. We continue to work to recover those last affected instances and volumes, and have opened notifications for the remaining impacted customers via the Personal Health Dashboard. For immediate recovery of those resources, we recommend replacing any remaining affected instances or volumes if possible.
- bawolff 5y ago
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- hyperman1 5y agoStrange to see them using PDT as a time zone. Both customers and local personell would be better served with either UTc or the local time zone.
- deleted 5y ago[deleted]
- ThePowerOfFuet 5y agoGoogle does it too. 'Murica.
- benglish11 5y agoAWS does this with all of their outages, presumably because Amazon HQ is in Washington (PDT). I would think UTC would be the preferred "universal" time for communicating these kinds of incidents, because AWS customers span the globe.
- c_o_n_v_e_x 5y agoFormer controls system guy and have worked in data centers. I'd be concerned about why a control system failure took down multiple air handlers. Units typically have their own controllers and can be configured to run by themselves without input from a "parent" controller.