5 ms·
6:54 PM PDT Starting at 1:18 PM PDT we experienced connectivity issues to some EC2 instances, increased API errors rates, and degraded performance for some EBS
by tyingq 5y ago
6:54 PM PDT
Starting at 1:18 PM PDT we experienced connectivity issues to some EC2 instances, increased API errors rates, and degraded performance for some EBS volumes within a single Availability Zone in the EU-CENTRAL-1 Region.
At 4:26 PM PDT, network connectivity was restored and the majority of affected instances and EBS volumes began to recover.
At 4:33 PM PDT, increased API error rates and latencies had also returned to normal levels. The issue has been resolved and the service is operating normally. The root cause of this issue was a failure of a control system which disabled multiple air handlers in the affected Availability Zone.
These air handlers move cool air to the servers and equipment, and when they were disabled, ambient temperatures began to rise. Servers and networking equipment in the affected Availability Zone began to power-off when unsafe temperatures were reached. Unfortunately, because this issue impacted several redundant network switches, a larger number of EC2 instances in this single Availability Zone lost network connectivity.
While our operators would normally had been able to restore cooling before impact, a fire suppression system activated inside a section of the affected Availability Zone. When this system activates, the data center is evacuated and sealed, and a chemical is dispersed to remove oxygen from the air to extinguish any fire. In order to recover the impacted instances and network equipment, we needed to wait until the fire department was able to inspect the facility.
After the fire department determined that there was no fire in the data center and it was safe to return, the building needed to be re-oxygenated before it was safe for engineers to enter the facility and restore the affected networking gear and servers. The fire suppression system that activated remains disabled. This system is designed to require smoke to activate and should not have discharged. This system will remain inactive until we are able to determine what triggered it improperly.
In the meantime, alternate fire suppression measures are being used to protect the data center. Once cooling was restored and the servers and network equipment was re-powered, affected instances recovered quickly. A very small number of remaining instances and volumes that were adversely affected by the increased ambient temperatures and loss of power remain unresolved.
We continue to work to recover those last affected instances and volumes, and have opened notifications for the remaining impacted customers via the Personal Health Dashboard. For immediate recovery of those resources, we recommend replacing any remaining affected instances or volumes if possible.
- bawolff 5y agoWow, sounds like the start of some implausible hollywood heist movie.
- bombcar 5y agoGordon, we cannot predict how long the system can operate at this level, nor how long the reading will take. Please work as quickly as you can. Uh...it's probably not a problem...probably...but I'm showing a small discrepancy in...well, no, it's well within acceptable bounds again. Sustaining sequence.
- dharmab 5y agoIn Tenet, the protagonist and his partner trigger a halon system in a building to force an evacuation as part of a heist.
- philjohn 5y agoI remember the summer job I had working in the European HQ of a chemical company. I was managing servers and thus had access to the server room - after the stern safety lecture about the Halon suppression system and being informed that "if the alarm goes off, don't even bother going for the respirator on the wall, just bolt it out of there double quick".
- toyg 5y agoIn the first season of Mr Robot, when they want to create financial chaos by wiping records of all debt, iirc they trigger a fire alert in a datacenter to force a backup wipe of some sort.
- freeflight 5y agoThey pretty much ransomwared all the digital records on the E Corp servers by encrypting them. The paper records they destroyed by blowing up a whole building, with hacked UPS systems. [0] [0] https://www.geekwire.com/2016/mr-robot-rewind-pwning-battery-backups-explosive-season-finale/ https://www.geekwire.com/2016/mr-robot-rewind-pwning-battery...