3 ms·

Is there a reason it often seems like backup generators fail? Is it that we don't hear the success stories of all the times they don't fail? Just due to not b

Is there a reason it often seems like backup generators fail? Is it that we don't hear the success stories of all the times they don't fail? Just due to not being tested often?

dsr_ 7y ago
If you were buying datacenter space instead of cloud resources, you would expect each datacenter to report to you quarterly that they had run their backup generator tests and fixed any problems that occurred. You would also expect to be told in advance about network maintenance and other foreseeable service issues. Cloud vendors, of course, get to hide all that from you until something goes wrong.
extrapickles 7y ago
Yes, you don’t hear the hundred or two times they start correctly, as they typically are tested quarterly to monthly. A good test is where you island the datacenter (run it only off the generators) so you can also make sure all the switchgear works.
toast0 7y ago
We generally only hear about the failures, but it's also a tricky thing to test. Simple setups won't put load on the generator during the periodic tests, which can result in outages if the generator will start, but can't run the load for whatever reason (ex: mechanical problems, or load size grew beyond capacity). More complicated setups may be able to switch the load to the generator, but not switch back to utility fast enough in case the generator under test fails during the test. The transfer switches themselves are prone to failures and hard to make redundant. It seems that it's pretty hard to get this right from the beginning too, every large datacenter ends up learning this again after they have a sequence of power incidents. That said, successful switch to generator for 1 hour and then generator failure is not a terrible outcome; if there was a notification, that's enough time to evacuate critical systems (assuming you have a plan).

Is there a reason it often seems like backup generators fail? Is it that we don't hear the success stories of all the times they don't fail? Just due to not being tested often?

dsr_ 7y ago

If you were buying datacenter space instead of cloud resources, you would expect each datacenter to report to you quarterly that they had run their backup generator tests and fixed any problems that occurred. You would also expect to be told in advance about network maintenance and other foreseeable service issues. Cloud vendors, of course, get to hide all that from you until something goes wrong.

extrapickles 7y ago

Yes, you don’t hear the hundred or two times they start correctly, as they typically are tested quarterly to monthly. A good test is where you island the datacenter (run it only off the generators) so you can also make sure all the switchgear works.

toast0 7y ago

We generally only hear about the failures, but it's also a tricky thing to test. Simple setups won't put load on the generator during the periodic tests, which can result in outages if the generator will start, but can't run the load for whatever reason (ex: mechanical problems, or load size grew beyond capacity). More complicated setups may be able to switch the load to the generator, but not switch back to utility fast enough in case the generator under test fails during the test. The transfer switches themselves are prone to failures and hard to make redundant. It seems that it's pretty hard to get this right from the beginning too, every large datacenter ends up learning this again after they have a sequence of power incidents. That said, successful switch to generator for 1 hour and then generator failure is not a terrible outcome; if there was a notification, that's enough time to evacuate critical systems (assuming you have a plan).