10 ms·

Sounds like Amazon is doing something wrong, shouldn't it fail over to Battery then Generator?

WestCoastJustin 14y ago
You're correct. It goes battery then generator. If they didn't use battery first, then when the power initial failed all systems would be off-line as the generator takes about 15-30 seconds to kick in.
- ericabiz 14y ago
  He's correct or incorrect, and that entirely depends on the facility. Many newer datacenters don't use battery banks--they are expensive to maintain and often cause more failures than they prevent.
rdoherty 14y ago
From what I can remember of a datacenter tour, most generators supply power to the batteries, which then supply power to servers. The batteries can only supply a few minutes (I think) of power, so the generators need to turn on immediately.
mrkurt 14y ago
Battery backup isn't usually considered a "failover" step, just like your desktop battery backup doesn't actually do much other than stop charging when the power goes out. Datacenters only really have battery backup to let the emergency generators come up.
ericabiz 14y ago
The batteries at colocation facilities are only designed to hold power long enough to transfer to the generator. They're also a huge single point of failure. A better design is a flywheel that generates enough power. But datacenters are often hit with these generator failures (in my experience, once every year or so.) Amazon had a correct setup--but not great testing. By the way, these are great questions to ask of your datacenter provider: Are there two completely redundant power systems up to and including the PDUs and generators? How often are those tested? How do I set up my servers properly so that if one circuit/PDU/generator fails, I don't lose power? There is a "right way" to do this--multiple power supplies in every server connected to 2 PDUs connected to 2 different generators--but it's expensive, and many/most low-end hosting providers won't set this up due to the cost. (I ran a colocation/dedicated server company from 2001-2007.)
- mjwalshe 14y ago
  no doing it right = everything runs off battery and you should be testing your gensets every day or so - we ceratinly did back in the 80's at Telecom Gold
- WestCoastJustin 14y ago
  Totally correct -- In our data center, and many of the universities that we work with, will have a right hand (RH) and left hand (LH) power feed for a dual PDU server, switch, router, console server, etc. You will typically run a power bar on the right and left hand side of the rack and wire then accordingly. You will have dedicated beakers, panels, UPS, and generator, etc for the RH/LH side. If you ever need to service a panel, then you can safely cut power knowing that everything will still be powered by its partner. This happens once and a while and allows you breathing room if you need to replace a breaker or a power feed fails. We also test our generators on a monthly cycle to test for failures. I also wanted to address you point about batteries. We have a device on each battery that monitors it's state. So we can find faults before they cause the entire UPS to fail.
  - bigiain 14y ago
    "We also test our generators on a monthly cycle to test for failures." Curious - when "testing" them, how long do you run them for and at what load? I could see the beancounters being _very_ unhappy with the ops people saying "we want to run both gen sets at full datacenter load for more than 10 minutes at a time, every month", which is what Amazon would have to have done to detect the faulty cooling fan problem. I'm guessing there are _some_ organisations who do that, but I suspect most datacenters don't.
    - deleted 14y ago
      [deleted]
    - WestCoastJustin 14y ago
      I work for the Fed. We are a remote site and have several power outages each year due to trees on the power lines or snow related issues. It's pretty much required and we've had zero issues justifying it.

You're correct. It goes battery then generator. If they didn't use battery first, then when the power initial failed all systems would be off-line as the generator takes about 15-30 seconds to kick in.

ericabiz 14y ago
He's correct or incorrect, and that entirely depends on the facility. Many newer datacenters don't use battery banks--they are expensive to maintain and often cause more failures than they prevent.

rdoherty 14y ago

From what I can remember of a datacenter tour, most generators supply power to the batteries, which then supply power to servers. The batteries can only supply a few minutes (I think) of power, so the generators need to turn on immediately.

mrkurt 14y ago

Battery backup isn't usually considered a "failover" step, just like your desktop battery backup doesn't actually do much other than stop charging when the power goes out. Datacenters only really have battery backup to let the emergency generators come up.

ericabiz 14y ago

The batteries at colocation facilities are only designed to hold power long enough to transfer to the generator. They're also a huge single point of failure. A better design is a flywheel that generates enough power. But datacenters are often hit with these generator failures (in my experience, once every year or so.) Amazon had a correct setup--but not great testing. By the way, these are great questions to ask of your datacenter provider: Are there two completely redundant power systems up to and including the PDUs and generators? How often are those tested? How do I set up my servers properly so that if one circuit/PDU/generator fails, I don't lose power? There is a "right way" to do this--multiple power supplies in every server connected to 2 PDUs connected to 2 different generators--but it's expensive, and many/most low-end hosting providers won't set this up due to the cost. (I ran a colocation/dedicated server company from 2001-2007.)

mjwalshe 14y ago
no doing it right = everything runs off battery and you should be testing your gensets every day or so - we ceratinly did back in the 80's at Telecom Gold
WestCoastJustin 14y ago
Totally correct -- In our data center, and many of the universities that we work with, will have a right hand (RH) and left hand (LH) power feed for a dual PDU server, switch, router, console server, etc. You will typically run a power bar on the right and left hand side of the rack and wire then accordingly. You will have dedicated beakers, panels, UPS, and generator, etc for the RH/LH side. If you ever need to service a panel, then you can safely cut power knowing that everything will still be powered by its partner. This happens once and a while and allows you breathing room if you need to replace a breaker or a power feed fails. We also test our generators on a monthly cycle to test for failures. I also wanted to address you point about batteries. We have a device on each battery that monitors it's state. So we can find faults before they cause the entire UPS to fail.
- bigiain 14y ago
  "We also test our generators on a monthly cycle to test for failures." Curious - when "testing" them, how long do you run them for and at what load? I could see the beancounters being _very_ unhappy with the ops people saying "we want to run both gen sets at full datacenter load for more than 10 minutes at a time, every month", which is what Amazon would have to have done to detect the faulty cooling fan problem. I'm guessing there are _some_ organisations who do that, but I suspect most datacenters don't.
  - deleted 14y ago
    [deleted]
  - WestCoastJustin 14y ago
    I work for the Fed. We are a remote site and have several power outages each year due to trees on the power lines or snow related issues. It's pretty much required and we've had zero issues justifying it.