From back in my sysadmin days when I worked for a Electrical Engineering firm which specialized in power distribution (power lines, power plants, etc). We had a

From back in my sysadmin days when I worked for a Electrical Engineering firm which specialized in power distribution (power lines, power plants, etc). We had a homegrown datacenter in the basement with dedicated AC, backups, and fire suppression system. It even had a home designed air handler system for moving vast amounts of air through the room in the event of AC failure. Oh and a home grown diesel generator -- cuz electrical engineers. Datacenter had about ~300 servers in it. Not huge, but not small either. The lynchpin in the system is this: neither the battery supplies or generator can run the AC or air handler, so when the power is out everything non-essential needs to come down to maintain sane temps in the DC. Anyway, my page goes off in the middle of the night -- power outage. The DC is running on battery backup. I hurry into the office to start powering things down as temps are climbing. I start shutting down VM's, blades, and 1/2U servers. About 1/2 of the way through, the power comes back on -- but the AC isn't kicking on (red flag). The air handler will function though, so let's run with that until the AC guy comes out. I start powering everything back up. At this point, a few co-workers trickle in to help. After about 2 minutes the fire suppression alarm triggers -- 30 sec to evacuate the DC. I glance over to the air handler vent, and it's SHOOTING flames into the DC. We oh-crap the heck out of there just in time to see the suppression system trigger and the door lock closed. I run to the electrical panel and kill the power to the AC and air handler knowing that they were possible sources of the fire. The fire dept. arrives and forces us out of the building. At this point, nearly the entire DC is cranking on sustainable power with 0 cooling. It's a locked box effectively. We watch our notification slowly alert to servers going down hard due to heat one-by-one. VM hosts -- boom. Network switches -- Boom. SAN -- BOOM. Long story short, we lost a number of servers and restored a lot of data from backup once things were back online. The cause was traced back to the wiring of the air handler motor. When the power came back on, only 2 of the 3 phases came back online. This was enough for the UPS system to operate, but not enough for the AC (wired correctly). The motor on the air handler was 3 phase but installed incorrectly (or something to that effect, it's been years and I'm not an electrician) allowing it to run, but turning it into a ticking time bomb of an electrical fire. Fun times.