5 ms·

Nice reminder about a "glitch" happening in one of the datacenters use at the airline I used to work. Went like this: Guy who shows around the new datacenter/o

Nice reminder about a "glitch" happening in one of the datacenters use at the airline I used to work. Went like this: Guy who shows around the new datacenter/ops guy demonstrated how the emergency power off works by lifting the protection plate. Protection plate unhinges suddenly and droppes onto emergency power off button. Hilarity ensues.

ianhawes 11y ago
I remember one of the LiveJournal outages (IIRC) was caused by a dude in the datacenter thinking that the emergency power off button opened one of the doors.
- misterbwong 11y ago
  Had a similar experience at an old company. One of the light switches in server room was placed next to a switch connected to the UPS of certain servers (I have no idea how/why this even came about). Confused the heck out of us when we were trying to figure out why some of our servers went on UPS power randomly at night. Turns out we'd get the notifications on nights the cleaning crew decided to flick the lights off then on again.
js2 11y ago
At former managed-hosting startup, we would deploy a pair of EMC symmetrixes into our DC cage as storage for our DB tier, with volumes mirrored over the pair. While equipment was being moved into the cage one day, someone accidentally banged the power-switch protection plate on one of the EMCs. Fortunately the power switch had to be held depressed for several seconds to turn off the equipment. Unfortunately, the protection plate itself got jammed in against power switch keeping it depressed. Hoisted by its own petard, as it were. Good thing we deployed those EMCs in pairs. But then curiosity gets the better of a DC ops guy wondering how the plate got jammed. So he punches the plate on the 2nd EMC causing it to similarly jam and power-off. Doh. When we told EMC about this flaw in the design, they deployed a fix to the production line - a rubber stopper under the plate next to the switch to protect the switch from the protection plate.
- coob 11y ago
  This is what happens when Homer Simpson works at your datacentre.
bitwize 11y ago
A data center where I work had a naked big red switch without a protection plate. Until I flagged up this lack and let the IT people know about it, it was a disaster waiting to happen. There's a plate on the switch now, praise Eris.
- geek_slop 11y ago
  Similar situation here. A big red button on the wall in the basement data center - nobody knew what it was for and were afraid to turn it off. One day a night operator decided he was going to flip the switch and see what happened. It cut power to the entire datacenter, including the elevator systems. We spent a few days getting all the systems back online. Had another instance where a pipe ran from the floor above right over the UPS. Construction workers on the first floor decided to poor some unusued paint into it (figured it was a drain pipe) and of course, we lost power again.
- philipw 11y ago
  I've seen many places where "the solution" to the unprotected EPO button was just a simple plastic cup!

ianhawes 11y ago

I remember one of the LiveJournal outages (IIRC) was caused by a dude in the datacenter thinking that the emergency power off button opened one of the doors.

misterbwong 11y ago
Had a similar experience at an old company. One of the light switches in server room was placed next to a switch connected to the UPS of certain servers (I have no idea how/why this even came about). Confused the heck out of us when we were trying to figure out why some of our servers went on UPS power randomly at night. Turns out we'd get the notifications on nights the cleaning crew decided to flick the lights off then on again.

js2 11y ago

At former managed-hosting startup, we would deploy a pair of EMC symmetrixes into our DC cage as storage for our DB tier, with volumes mirrored over the pair. While equipment was being moved into the cage one day, someone accidentally banged the power-switch protection plate on one of the EMCs. Fortunately the power switch had to be held depressed for several seconds to turn off the equipment. Unfortunately, the protection plate itself got jammed in against power switch keeping it depressed. Hoisted by its own petard, as it were. Good thing we deployed those EMCs in pairs. But then curiosity gets the better of a DC ops guy wondering how the plate got jammed. So he punches the plate on the 2nd EMC causing it to similarly jam and power-off. Doh. When we told EMC about this flaw in the design, they deployed a fix to the production line - a rubber stopper under the plate next to the switch to protect the switch from the protection plate.

coob 11y ago
This is what happens when Homer Simpson works at your datacentre.

bitwize 11y ago

A data center where I work had a naked big red switch without a protection plate. Until I flagged up this lack and let the IT people know about it, it was a disaster waiting to happen. There's a plate on the switch now, praise Eris.

geek_slop 11y ago
Similar situation here. A big red button on the wall in the basement data center - nobody knew what it was for and were afraid to turn it off. One day a night operator decided he was going to flip the switch and see what happened. It cut power to the entire datacenter, including the elevator systems. We spent a few days getting all the systems back online. Had another instance where a pipe ran from the floor above right over the UPS. Construction workers on the first floor decided to poor some unusued paint into it (figured it was a drain pipe) and of course, we lost power again.
philipw 11y ago
I've seen many places where "the solution" to the unprotected EPO button was just a simple plastic cup!