4 ms·

Most of the time, it's not the actual generator that fails. It's often either the part that detects power failure (fluctuations, brownouts, ... are harder to de

Most of the time, it's not the actual generator that fails. It's often either the part that detects power failure (fluctuations, brownouts, ... are harder to detect than total failure) or the switchover hardware to the generator. If you want to switch 20KV, you're really handling an awful lot of energy. Physics kinda get in the way there. Switching paket based networks is often much easier because you can just hold on to the packet and retransmit if a line fails, that's why it's much rarer to have total network failures.

blattimwind 9y ago
To add to this, backup power systems are in fairly widespread deployment and bigger installations are tested and maintained on a schedule. You'll never see "hospital transferred emergency-available circuits to backup system; no malfunction" in the news, yet it happens every few weeks in every installation. Generators themselves are very reliable machines. The engines are cast iron (instead of cast aluminium as found in all cars since two decades or so) and are nowhere near as close to the edge of material capacity compared to car engines. Electrical generators are essentially just a big blob of metal, they are simple, efficient and very reliable technology giving many decades of service. Generator controllers are designed to be reliable and aren't terribly complicated, either.
- AstralStorm 9y ago
  Apparently OVH did test their generator system but the outage detector or controller failed.
  - exikyut 9y ago
    Hmm. Makes me wonder. If generators really are just a giant blob of metal and very simple (and I have no reason to disbelieve the GP comment)... well... it could be kind of interesting to build an open source software stack to handle switchover. Because, disclaimers notwithstanding, the code would ostensibly be super simple too. So even if it couldn't officially be used directly, it would certainly provide a good base for engineers to copy-paste, either literally or ideologically (and then thoroughly verify, of course!). Okay... thinking about it, I'm probably wrong - either generator control has some fundamental intricacies that make it not-completely-simple, or all sites have edge cases that have to be baked into the firmware by a system integrator/electrical engineer. I say this because I'm (genuinely) trying to figure out why the PLCs failed - both in OVH's case, and in AWS's case (see elsewhere in this thread, https://news.ycombinator.com/item?id=15676189 https://news.ycombinator.com/item?id=15676189). It's obvious there are crazy but legit reasons for this kind of thing to happen. I'm very curious what the complexity scale here is.
    - jotm 9y ago
      The actual diesel engines and generators are simple metal. Not the switching circuits. If the electric controllers can't handle the load/switching speed/whatever, no clever software and/or reliability in the actual generators will help.
      - exikyut 9y ago
        Ah. I've seen what switching 20kV looks like in videos - yeah, that kind of thing requires very careful design, and is invariably going to come entangled with a PLC-style controller, as is the norm for industrial equipment.
specialp 9y ago
We've had this happen in our small onsite datacenter. A single phase out of the 3 changed waveform after a lightning strike on a utility pole. The power monitor to turn on the generator didn't see this a problem, but the UPS did. So the UPS was on for several minutes and I noticed some of the ceiling lights out that depended on that phase. Fortunately I and another employee were there and turned on the manual transfer to put us on generator power. Disaster backups are hard to test as to do it properly they have to be integration tested, and doing so might break production. Sure your generator might power on for 5 minutes a month and make people happy, but there are many unforeseen things that are hard to test for.
bluedino 9y ago
>> Most of the time, it's not the actual generator that fails. In our case, we had a generator run out of fuel. Another time we had one have the battery die (they're started by batteries just like a car engine)

blattimwind 9y ago

To add to this, backup power systems are in fairly widespread deployment and bigger installations are tested and maintained on a schedule. You'll never see "hospital transferred emergency-available circuits to backup system; no malfunction" in the news, yet it happens every few weeks in every installation. Generators themselves are very reliable machines. The engines are cast iron (instead of cast aluminium as found in all cars since two decades or so) and are nowhere near as close to the edge of material capacity compared to car engines. Electrical generators are essentially just a big blob of metal, they are simple, efficient and very reliable technology giving many decades of service. Generator controllers are designed to be reliable and aren't terribly complicated, either.

AstralStorm 9y ago
Apparently OVH did test their generator system but the outage detector or controller failed.
- exikyut 9y ago
  Hmm. Makes me wonder. If generators really are just a giant blob of metal and very simple (and I have no reason to disbelieve the GP comment)... well... it could be kind of interesting to build an open source software stack to handle switchover. Because, disclaimers notwithstanding, the code would ostensibly be super simple too. So even if it couldn't officially be used directly, it would certainly provide a good base for engineers to copy-paste, either literally or ideologically (and then thoroughly verify, of course!). Okay... thinking about it, I'm probably wrong - either generator control has some fundamental intricacies that make it not-completely-simple, or all sites have edge cases that have to be baked into the firmware by a system integrator/electrical engineer. I say this because I'm (genuinely) trying to figure out why the PLCs failed - both in OVH's case, and in AWS's case (see elsewhere in this thread, https://news.ycombinator.com/item?id=15676189 https://news.ycombinator.com/item?id=15676189). It's obvious there are crazy but legit reasons for this kind of thing to happen. I'm very curious what the complexity scale here is.
  - jotm 9y ago
    The actual diesel engines and generators are simple metal. Not the switching circuits. If the electric controllers can't handle the load/switching speed/whatever, no clever software and/or reliability in the actual generators will help.
    - exikyut 9y ago
      Ah. I've seen what switching 20kV looks like in videos - yeah, that kind of thing requires very careful design, and is invariably going to come entangled with a PLC-style controller, as is the norm for industrial equipment.

specialp 9y ago

We've had this happen in our small onsite datacenter. A single phase out of the 3 changed waveform after a lightning strike on a utility pole. The power monitor to turn on the generator didn't see this a problem, but the UPS did. So the UPS was on for several minutes and I noticed some of the ceiling lights out that depended on that phase. Fortunately I and another employee were there and turned on the manual transfer to put us on generator power. Disaster backups are hard to test as to do it properly they have to be integration tested, and doing so might break production. Sure your generator might power on for 5 minutes a month and make people happy, but there are many unforeseen things that are hard to test for.

bluedino 9y ago

>> Most of the time, it's not the actual generator that fails. In our case, we had a generator run out of fuel. Another time we had one have the battery die (they're started by batteries just like a car engine)