4 ms·

Great update thanks. Perhaps I am being naive, but I thought data centers had back-up power to cover this exact problem? I.e. battery power until the generato

Great update thanks. Perhaps I am being naive, but I thought data centers had back-up power to cover this exact problem? I.e. battery power until the generators kick in... or is this how it used to be done and these days no one bothers and just relies on fail-over to other sites????

pyvpx 10y ago
despite best efforts, sometimes the generators don't kick in. sometimes not all of them kick in. sometimes one ATS in one feed in one section that just happens to house one half of an important switch chassis doesn't work as it did during tests last quarter.
- snowy 10y ago
  True. But Data centers usually have an A and B power systems independent of each other for redundancy . Equipment is usually connected to both. The failure of either should not cause an outage. (Unless some one is stupid enough to connect core network equipment to only A or B power)
  - marcusr 10y ago
    Sometimes only A power or B power is delivered to the rack (otherwise you could receive different phases). Power failures also occur sometimes when the issue is on the wrong side of where power is distributed - we've lost power to our servers when maintenance has gone wrong for example.
    - snowy 10y ago
      From my point of view if you end up with mission critical equipment connected only to A or B power then its engineered wrong.
      - edwhitesell 10y ago
        Very true, but sometimes this happens at fault of the facility, not the affected company. I've been in a facility experiencing widespread power issues because customers were incorrectly delivered A+A or B+B power, instead of A+B.
    - Aloha 10y ago
      This is why I'm a proponent battery floated of DC Power for mission critical systems.
- chrishacken 10y ago
  Exactly. You can only test for these scenarios to a certain extent and even when everything works as expected during tests, things still tend to break during production when shit hits the fan.
pbhjpbhj 10y ago
Like you, I'm surprised. I thought major data centres had redundant power from the grid too (not just battery and generator), so the UPS smooths the power transition as the switching gear fails over to the alternate grid connection; then generators come up if that fails too? Is that considered over-engineering?
redcalx 10y ago
Perhaps a more serious point is that a single power failure (in one data center) can have such a wide impact.
thomseddon 10y ago
They usually do, but these only protect from grid failures. HEX certainly does have battery + generators (it's one of the most important DC's in the country). Components inside the DC should be redundant, but the customer can decide whether to have dual power feeds from redundant areas of the DC provided to their rack/cage. Even with all this, issues do happen, Telecity had a very high profile and prolonged issue at the end of last year that affected AWS & Azure amongst others: http://www.theregister.co.uk/2015/11/18/telecity_outage_fix_failed/ http://www.theregister.co.uk/2015/11/18/telecity_outage_fix_...

pyvpx 10y ago

despite best efforts, sometimes the generators don't kick in. sometimes not all of them kick in. sometimes one ATS in one feed in one section that just happens to house one half of an important switch chassis doesn't work as it did during tests last quarter.

snowy 10y ago
True. But Data centers usually have an A and B power systems independent of each other for redundancy . Equipment is usually connected to both. The failure of either should not cause an outage. (Unless some one is stupid enough to connect core network equipment to only A or B power)
- marcusr 10y ago
  Sometimes only A power or B power is delivered to the rack (otherwise you could receive different phases). Power failures also occur sometimes when the issue is on the wrong side of where power is distributed - we've lost power to our servers when maintenance has gone wrong for example.
  - snowy 10y ago
    From my point of view if you end up with mission critical equipment connected only to A or B power then its engineered wrong.
    - edwhitesell 10y ago
      Very true, but sometimes this happens at fault of the facility, not the affected company. I've been in a facility experiencing widespread power issues because customers were incorrectly delivered A+A or B+B power, instead of A+B.
  - Aloha 10y ago
    This is why I'm a proponent battery floated of DC Power for mission critical systems.
chrishacken 10y ago
Exactly. You can only test for these scenarios to a certain extent and even when everything works as expected during tests, things still tend to break during production when shit hits the fan.

pbhjpbhj 10y ago

Like you, I'm surprised. I thought major data centres had redundant power from the grid too (not just battery and generator), so the UPS smooths the power transition as the switching gear fails over to the alternate grid connection; then generators come up if that fails too? Is that considered over-engineering?

redcalx 10y ago

Perhaps a more serious point is that a single power failure (in one data center) can have such a wide impact.

thomseddon 10y ago

They usually do, but these only protect from grid failures. HEX certainly does have battery + generators (it's one of the most important DC's in the country). Components inside the DC should be redundant, but the customer can decide whether to have dual power feeds from redundant areas of the DC provided to their rack/cage. Even with all this, issues do happen, Telecity had a very high profile and prolonged issue at the end of last year that affected AWS & Azure amongst others: http://www.theregister.co.uk/2015/11/18/telecity_outage_fix_failed/ http://www.theregister.co.uk/2015/11/18/telecity_outage_fix_...