6 ms·

I work in the power generation industry. We have a lot of primary/secondary as well primary/standby systems. The cardinal rule is - test everything. Test it wh

I work in the power generation industry. We have a lot of primary/secondary as well primary/standby systems. The cardinal rule is - test everything. Test it when you install and commission it, test it when part of the system changes, test it periodically during maintenance shutdowns, test it when you have a convenient time to do so without losing production (such as an outage to a different system that necessitates your system being offline). Test components individually - at the factory and in the field - and test systems end-to-end. Test with the most adverse possible conditions, not the most optimal. Test beyond your normal operating envelope. When I participate in design reviews as a maintenance engineer my primary line of inquiry is: how will I test this stuff during operations? Has it been designed in a way that makes testing impossible without an expensive outage? Can workers safely gain access to components that need routine testing without having to de-energize other, unrelated parts of the system? Now not every industry has the budget available to test to the extent that we do. But even in our industry, I often see a penny-wise, pound-foolish approach of "assume this thing works perfectly from the factory then act surprised years later when an abnormal condition occurs and it fails". If something was not tested, it may not work. It might be for a very simple reason (like mis-configuration of a single setting during installation) or a very complex reason. But either way, wouldn't the company rather know their equipment doesn't work -before- putting it into service than after?

a1369209993 6y ago
Apropos of this, in TFA's case they apparently never tested "rip the battery out while the system is running". (There are other, harder-to-test ways for a battery to fail, but that seems like a obvious and easily-tested failure mode.)
kitteh 6y ago
The telco Central Office I worked in had weekly tests where we'd run the generator for an hour. Everyone's desk computer required a UPS as they were on utility. Comically after a few years you'd have to get a new UPS at your desk as theyd die from continual use.
- myself248 6y ago
  Most of the COs I worked in did monthly generator "dry runs" without putting load on it, and then every 6 months they'd do a "drawdown test": Disable the turbine. Kill AC power (turn off all the rectifiers and HVAC) and let the office run on battery for a while. Closely monitor the batteries, and the temperature in critical areas. After a good while on battery (an hour or two, if I recall), walk leisurely over to the turbine and enable it. Let it start and warm up, then transfer the rectifiers and HVAC over to generator power. Closely monitor the turbine while the batteries recharge, the HVAC while on unusual power, and the humans who are really glad to have air conditioning back. When satisfied that all is copacetic, transfer things back to utility AC, shut down the turbine, and call the fuel company to top off the tank. Most of the batteries in these offices were between 10 and 20 years old, with a couple instances of the beautiful cylindrical "Bell Cells" still in service. They were sized to not break a sweat running the whole building, and impeccably maintained, so they didn't tend to fail prematurely.
  - blaser-waffle 6y ago
    Same basic idea. We'd exercise the generators weekly/bi-weekly, depending on the site and configuration. Once a quarter we'd do a live load transfer, and watch all the PDUs and UPS systems do their thing. Those could be stressful...
laurent92 6y ago
How do you test diesel generators which back up a nuclear reactor? I understand you do it during the plant downtime, but don’t you require something at the other end of the network that consumes the 1GW you generate?
- solstice 6y ago
  Naive idea: Putting a big copper rod in whatever body of water the plant uses for cooling and connecting power to it should do it no?
  - Scoundreller 6y ago
    Not copper. You want resistance to be highest at your heat dump. Otherwise all your wires melt.
- deleted 6y ago
  [deleted]
- a9h74j 6y ago
  The diesel generators can back up power to safety systems, so closer to MWs than GWs. Some run with left-handed and right-handed engines coupled face-to-face for redundancy; thus you might AFAIK shut down fuel injection to one to prove the other is supplying torque.
- gh02t 6y ago
  A combination of routing the power elsewhere in the grid when possible and banks of gigantic (shipping container sized) dummy loads. As other people have noted, you don't need to dissipate the full amount of power all at once, you test the generators individually. http://resistive-loadbank.sell.everychina.com/p-109417602-3546kva-inductive-power-banks-for-generator-testing-dummy-load-bank.html http://resistive-loadbank.sell.everychina.com/p-109417602-35... is the first example I pulled up, and there are bigger ones.
- bluGill 6y ago
  You test them live: someone turns off the outside power switch and you watch them start and take over automatically. Everyone is watching the gauges and if there is a sign of trouble you reengage mains power, then fix the problem and retry. This test is done monthly. I don't have actual knowledge of nuclear power plants, that is classified beyond the public. However hospitals do the above, because if thing dont work on a bright sunny day how do you think they will work when a tornado has not only taken out power, but also filled the er with server trauma cases.
mprovost 6y ago
That's a good point about testing. I think the key point is where the testing is happening. I'm sure the UPS (and other power equipment) is tested by the manufacturer for many different failure scenarios. The issue is that to the end user, the response is often surprising. So I'm sure to the APC engineer, having the unit shut down when the battery fails (according to some metric) but while there is still mains power available is something that they tested. But the consumer hasn't tested a battery failure, and in fact it may be difficult to test the exact failure mode (low voltage etc) without access to some spare, old, batteries. So when it happens and you're surprised and you call the manufacturer, they (almost always) will say that it operated according to how it was designed and tested. It just didn't do what you wanted or expected. At least that was always my experience doing a post mortem on a datacentre outage.

a1369209993 6y ago

Apropos of this, in TFA's case they apparently never tested "rip the battery out while the system is running". (There are other, harder-to-test ways for a battery to fail, but that seems like a obvious and easily-tested failure mode.)

kitteh 6y ago

The telco Central Office I worked in had weekly tests where we'd run the generator for an hour. Everyone's desk computer required a UPS as they were on utility. Comically after a few years you'd have to get a new UPS at your desk as theyd die from continual use.

myself248 6y ago
Most of the COs I worked in did monthly generator "dry runs" without putting load on it, and then every 6 months they'd do a "drawdown test": Disable the turbine. Kill AC power (turn off all the rectifiers and HVAC) and let the office run on battery for a while. Closely monitor the batteries, and the temperature in critical areas. After a good while on battery (an hour or two, if I recall), walk leisurely over to the turbine and enable it. Let it start and warm up, then transfer the rectifiers and HVAC over to generator power. Closely monitor the turbine while the batteries recharge, the HVAC while on unusual power, and the humans who are really glad to have air conditioning back. When satisfied that all is copacetic, transfer things back to utility AC, shut down the turbine, and call the fuel company to top off the tank. Most of the batteries in these offices were between 10 and 20 years old, with a couple instances of the beautiful cylindrical "Bell Cells" still in service. They were sized to not break a sweat running the whole building, and impeccably maintained, so they didn't tend to fail prematurely.
- blaser-waffle 6y ago
  Same basic idea. We'd exercise the generators weekly/bi-weekly, depending on the site and configuration. Once a quarter we'd do a live load transfer, and watch all the PDUs and UPS systems do their thing. Those could be stressful...

laurent92 6y ago

How do you test diesel generators which back up a nuclear reactor? I understand you do it during the plant downtime, but don’t you require something at the other end of the network that consumes the 1GW you generate?

solstice 6y ago
Naive idea: Putting a big copper rod in whatever body of water the plant uses for cooling and connecting power to it should do it no?
- Scoundreller 6y ago
  Not copper. You want resistance to be highest at your heat dump. Otherwise all your wires melt.
deleted 6y ago
[deleted]
a9h74j 6y ago
The diesel generators can back up power to safety systems, so closer to MWs than GWs. Some run with left-handed and right-handed engines coupled face-to-face for redundancy; thus you might AFAIK shut down fuel injection to one to prove the other is supplying torque.
gh02t 6y ago
A combination of routing the power elsewhere in the grid when possible and banks of gigantic (shipping container sized) dummy loads. As other people have noted, you don't need to dissipate the full amount of power all at once, you test the generators individually. http://resistive-loadbank.sell.everychina.com/p-109417602-3546kva-inductive-power-banks-for-generator-testing-dummy-load-bank.html http://resistive-loadbank.sell.everychina.com/p-109417602-35... is the first example I pulled up, and there are bigger ones.
bluGill 6y ago
You test them live: someone turns off the outside power switch and you watch them start and take over automatically. Everyone is watching the gauges and if there is a sign of trouble you reengage mains power, then fix the problem and retry. This test is done monthly. I don't have actual knowledge of nuclear power plants, that is classified beyond the public. However hospitals do the above, because if thing dont work on a bright sunny day how do you think they will work when a tornado has not only taken out power, but also filled the er with server trauma cases.

mprovost 6y ago

That's a good point about testing. I think the key point is where the testing is happening. I'm sure the UPS (and other power equipment) is tested by the manufacturer for many different failure scenarios. The issue is that to the end user, the response is often surprising. So I'm sure to the APC engineer, having the unit shut down when the battery fails (according to some metric) but while there is still mains power available is something that they tested. But the consumer hasn't tested a battery failure, and in fact it may be difficult to test the exact failure mode (low voltage etc) without access to some spare, old, batteries. So when it happens and you're surprised and you call the manufacturer, they (almost always) will say that it operated according to how it was designed and tested. It just didn't do what you wanted or expected. At least that was always my experience doing a post mortem on a datacentre outage.