6 ms·

Why are backup generators so unreliable? And that is unreliable by the standard of a non-critical system let alone that of an emergency system where reliability

Why are backup generators so unreliable? And that is unreliable by the standard of a non-critical system let alone that of an emergency system where reliability is their sole purpose. Is it just a (non)-survivorship bias where we most commonly talk about the failure cases and they actually have a stellar 99.99999% record that doesn't make headlines?

cjg_ 9y ago
In this case the generators themselves worked, just not the mechanism to cut over to them.
- duncanawoods 9y ago
  I include that bit of the system in "backup generator" to distinguish it from "generator".
- tyingq 9y ago
  Generally called a "static switch", and they are prone to issues. Here's one: https://www.aegps.com/en/products/ups-accessories/static-transfer-switch/ https://www.aegps.com/en/products/ups-accessories/static-tra...
  - dx034 9y ago
    Are they automatically a single point of failure or can you install several of those on one site?
Xylakant 9y ago
Most of the time, it's not the actual generator that fails. It's often either the part that detects power failure (fluctuations, brownouts, ... are harder to detect than total failure) or the switchover hardware to the generator. If you want to switch 20KV, you're really handling an awful lot of energy. Physics kinda get in the way there. Switching paket based networks is often much easier because you can just hold on to the packet and retransmit if a line fails, that's why it's much rarer to have total network failures.
- blattimwind 9y ago
  To add to this, backup power systems are in fairly widespread deployment and bigger installations are tested and maintained on a schedule. You'll never see "hospital transferred emergency-available circuits to backup system; no malfunction" in the news, yet it happens every few weeks in every installation. Generators themselves are very reliable machines. The engines are cast iron (instead of cast aluminium as found in all cars since two decades or so) and are nowhere near as close to the edge of material capacity compared to car engines. Electrical generators are essentially just a big blob of metal, they are simple, efficient and very reliable technology giving many decades of service. Generator controllers are designed to be reliable and aren't terribly complicated, either.
  - AstralStorm 9y ago
    Apparently OVH did test their generator system but the outage detector or controller failed.
    - exikyut 9y ago
      Hmm. Makes me wonder. If generators really are just a giant blob of metal and very simple (and I have no reason to disbelieve the GP comment)... well... it could be kind of interesting to build an open source software stack to handle switchover. Because, disclaimers notwithstanding, the code would ostensibly be super simple too. So even if it couldn't officially be used directly, it would certainly provide a good base for engineers to copy-paste, either literally or ideologically (and then thoroughly verify, of course!). Okay... thinking about it, I'm probably wrong - either generator control has some fundamental intricacies that make it not-completely-simple, or all sites have edge cases that have to be baked into the firmware by a system integrator/electrical engineer. I say this because I'm (genuinely) trying to figure out why the PLCs failed - both in OVH's case, and in AWS's case (see elsewhere in this thread, https://news.ycombinator.com/item?id=15676189 https://news.ycombinator.com/item?id=15676189). It's obvious there are crazy but legit reasons for this kind of thing to happen. I'm very curious what the complexity scale here is.
sphix0r 9y ago
Similar to many of us make backups in IT but how many actually do try them out and restore?
- scurvy 9y ago
  waves hand Every day. All of them (automated).
  - darklajid 9y ago
    Can you turn/expand that into a somewhat longer answer, a list of links that you perused to build that or .. a post on its own?
    - scurvy 9y ago
      Brains and know-how my good man, not copypasta.
- msh 9y ago
  We have yearly recovery exercises for randomly chosen subsets of servers and databases.
delsarto 9y ago
I think they're just large, complicated physical things where stuff goes wrong, even if you're maintaining and testing. My best story was as a young student turning up for my helpdesk shift at about 5:50am to a phalanx of fire engines and the Hazmat team. The generator in the basement was maintained and tested. This time it had started when power went out, but there was a pump that filled a holding tank from a 10,000ish litre tank under the building (speced to run for a week apparently). A pipe had cracked, and so it just kept pumping fuel on to the floor, carpark and into the local stream ... The only person in the building overnight was the operator (tape changer and report runner) some 5 stories above all this, who eventually smelt it and eventually investgated. It was a massive nightmare, and the place stank for weeks. Then there was the time the halon system went off unnecessarily...
- _nalply 9y ago
  I have a similar story of a smallish colocation site located in the cellar of a high school building in Switzerland where we hosted two servers for a small company ten years ago. The operator company had a generator at the site. During a thunderstorm there was a power outage, so at first all was fine. However a lightning nearby destroyed some critical circuitry of the generator. I additionally made the mistake of putting the watchdog system in the same colocation so that I didn't even get a notification about the outage. But after all I would have been powerless anyway.
- robryk 9y ago
  The particular failure mode you're talking about is reasonably easy to reliably remove: all fuel piping should be under lower than atmospheric pressure. You accomplish this by installing suction pumps next to consumers (and install additional pumps if the height difference is too large). That way, if a pipe breaks (except for the short pieces of pipe between the pipe and consumer), there will be no sustained fuel leak.
  - Johnny555 9y ago
    Building constraints don't always make it that easy -- the generator could be below the tank level, so siphon action can still siphon the tank empty, or it can be a few floors about the tank, too high to suck fuel from the tank so it has to be pushed up from below.
deleted 9y ago
[deleted]
batrat 9y ago
People. In my experience faulty generators come from bad maintenance. We have one in every site and 2 generators in locations where there is no electricity (mostly Caterpillar). They mostly fail due to lack of/bad maintenance. Like others said, yes they are big and with lots of moving parts. But if you take care of them (every month by the book) no one should have any unexpected problems. For me is mind blowing that OVH did the last power failure test in MAY. WHY so rare? IMO that is 1st mistake.
- malydok 9y ago
  > SBG's latest test for backup recovery were at the end of May 2017. During this last test, we powered SBG only from the generators for 8 hours without any issues and every month we test the backup generators empty. They do seem to test the generators every month.
staunch 9y ago
Because it's "backup" hardware. Like a lot of insurance, it's often just a false sense of security. It would be interesting to run a datacenter completely on batteries that are charged from solar + the grid + gas generators. You could run the generators at optimal power output and use them daily. The batteries required for ~24-48 hours might still be too expensive to make this possible. Maybe some kind of super low power datacenter could pull this off. One day it shall be mine.

cjg_ 9y ago

In this case the generators themselves worked, just not the mechanism to cut over to them.

duncanawoods 9y ago
I include that bit of the system in "backup generator" to distinguish it from "generator".
tyingq 9y ago
Generally called a "static switch", and they are prone to issues. Here's one: https://www.aegps.com/en/products/ups-accessories/static-transfer-switch/ https://www.aegps.com/en/products/ups-accessories/static-tra...
- dx034 9y ago
  Are they automatically a single point of failure or can you install several of those on one site?

Xylakant 9y ago

Most of the time, it's not the actual generator that fails. It's often either the part that detects power failure (fluctuations, brownouts, ... are harder to detect than total failure) or the switchover hardware to the generator. If you want to switch 20KV, you're really handling an awful lot of energy. Physics kinda get in the way there. Switching paket based networks is often much easier because you can just hold on to the packet and retransmit if a line fails, that's why it's much rarer to have total network failures.

blattimwind 9y ago
To add to this, backup power systems are in fairly widespread deployment and bigger installations are tested and maintained on a schedule. You'll never see "hospital transferred emergency-available circuits to backup system; no malfunction" in the news, yet it happens every few weeks in every installation. Generators themselves are very reliable machines. The engines are cast iron (instead of cast aluminium as found in all cars since two decades or so) and are nowhere near as close to the edge of material capacity compared to car engines. Electrical generators are essentially just a big blob of metal, they are simple, efficient and very reliable technology giving many decades of service. Generator controllers are designed to be reliable and aren't terribly complicated, either.
- AstralStorm 9y ago
  Apparently OVH did test their generator system but the outage detector or controller failed.
  - exikyut 9y ago
    Hmm. Makes me wonder. If generators really are just a giant blob of metal and very simple (and I have no reason to disbelieve the GP comment)... well... it could be kind of interesting to build an open source software stack to handle switchover. Because, disclaimers notwithstanding, the code would ostensibly be super simple too. So even if it couldn't officially be used directly, it would certainly provide a good base for engineers to copy-paste, either literally or ideologically (and then thoroughly verify, of course!). Okay... thinking about it, I'm probably wrong - either generator control has some fundamental intricacies that make it not-completely-simple, or all sites have edge cases that have to be baked into the firmware by a system integrator/electrical engineer. I say this because I'm (genuinely) trying to figure out why the PLCs failed - both in OVH's case, and in AWS's case (see elsewhere in this thread, https://news.ycombinator.com/item?id=15676189 https://news.ycombinator.com/item?id=15676189). It's obvious there are crazy but legit reasons for this kind of thing to happen. I'm very curious what the complexity scale here is.

sphix0r 9y ago

Similar to many of us make backups in IT but how many actually do try them out and restore?

scurvy 9y ago
waves hand Every day. All of them (automated).
- darklajid 9y ago
  Can you turn/expand that into a somewhat longer answer, a list of links that you perused to build that or .. a post on its own?
  - scurvy 9y ago
    Brains and know-how my good man, not copypasta.
msh 9y ago
We have yearly recovery exercises for randomly chosen subsets of servers and databases.

delsarto 9y ago

I think they're just large, complicated physical things where stuff goes wrong, even if you're maintaining and testing. My best story was as a young student turning up for my helpdesk shift at about 5:50am to a phalanx of fire engines and the Hazmat team. The generator in the basement was maintained and tested. This time it had started when power went out, but there was a pump that filled a holding tank from a 10,000ish litre tank under the building (speced to run for a week apparently). A pipe had cracked, and so it just kept pumping fuel on to the floor, carpark and into the local stream ... The only person in the building overnight was the operator (tape changer and report runner) some 5 stories above all this, who eventually smelt it and eventually investgated. It was a massive nightmare, and the place stank for weeks. Then there was the time the halon system went off unnecessarily...

_nalply 9y ago
I have a similar story of a smallish colocation site located in the cellar of a high school building in Switzerland where we hosted two servers for a small company ten years ago. The operator company had a generator at the site. During a thunderstorm there was a power outage, so at first all was fine. However a lightning nearby destroyed some critical circuitry of the generator. I additionally made the mistake of putting the watchdog system in the same colocation so that I didn't even get a notification about the outage. But after all I would have been powerless anyway.
robryk 9y ago
The particular failure mode you're talking about is reasonably easy to reliably remove: all fuel piping should be under lower than atmospheric pressure. You accomplish this by installing suction pumps next to consumers (and install additional pumps if the height difference is too large). That way, if a pipe breaks (except for the short pieces of pipe between the pipe and consumer), there will be no sustained fuel leak.
- Johnny555 9y ago
  Building constraints don't always make it that easy -- the generator could be below the tank level, so siphon action can still siphon the tank empty, or it can be a few floors about the tank, too high to suck fuel from the tank so it has to be pushed up from below.

deleted 9y ago

[deleted]

batrat 9y ago

People. In my experience faulty generators come from bad maintenance. We have one in every site and 2 generators in locations where there is no electricity (mostly Caterpillar). They mostly fail due to lack of/bad maintenance. Like others said, yes they are big and with lots of moving parts. But if you take care of them (every month by the book) no one should have any unexpected problems. For me is mind blowing that OVH did the last power failure test in MAY. WHY so rare? IMO that is 1st mistake.

malydok 9y ago
> SBG's latest test for backup recovery were at the end of May 2017. During this last test, we powered SBG only from the generators for 8 hours without any issues and every month we test the backup generators empty. They do seem to test the generators every month.

staunch 9y ago

Because it's "backup" hardware. Like a lot of insurance, it's often just a false sense of security. It would be interesting to run a datacenter completely on batteries that are charged from solar + the grid + gas generators. You could run the generators at optimal power output and use them daily. The batteries required for ~24-48 hours might still be too expensive to make this possible. Maybe some kind of super low power datacenter could pull this off. One day it shall be mine.