12 ms·

This reminded me of James Hamilton's (the AWS datacentre guru) writeup of the power outage at the 2013 SuperBowl. Specifically how the default configuration of

This reminded me of James Hamilton's (the AWS datacentre guru) writeup of the power outage at the 2013 SuperBowl. Specifically how the default configuration of backup generators is often to protect themselves, even if that means shutting down the things they are meant to keep running. In my own experience running a reasonably sized datacentre I was often surprised at how UPSs and generators reacted to adverse conditions. One thing with power that you don't want is to be surprised. But it's difficult to test this equipment so it's mostly a learn as you go experience. https://perspectives.mvdirona.com/2013/02/the-power-failure-seen-around-the-world/ https://perspectives.mvdirona.com/2013/02/the-power-failure-...

generatorguy 6y ago
Electrical Engineers would determine the settings for the electrical protection of a generator. Mechanical Engineers would determine the settings for mechanical protection of a motor. To operate the equipment beyond thresholds would result in reduced lifespan or immediate and permanent damage to the equipment. If this is what the customer requires they have to specify that to the Engineer so that when the equipment blows up the Engineer is not liable. Diesel generators for hospitals and water pumps for fire suppression systems are generally set up with very loose settings as it is clearly worse for a patient to die or a building to burn down than to destroy a machine, and this requirement is in the specifications.
- brudgers 6y ago
  Diesel generators for hospitals and water pumps for fire suppression systems Those systems also require regular testing by qualified personnel. NFPA 25 requires monthly testing of fire pumps and annual flow testing. For low rise buildings with inadequate municipal water pressure, a water tower might be cheaper and easier. NFPA 99 - Standard for health care facilities, requires emergency generators to be tested 12 times per year.
  - generatorguy 6y ago
    The testing would consist of running the diesel generators under load but not operating in abnormal conditions.
    - bluGill 6y ago
      True, but the easiest test is to pull the main power to the hospital and watch the system automatically take over. If anything fails you know what is wrong. This is scheduled so that those in surgery (or other worrisome situations) have extra help and can signal get back to mains power now.
      - generatorguy 6y ago
        ideally the hospital’s generator control system or transfer switch is able to start and synchronize the generators and turn up the throttle until no power is being imported from the utility prior to opening the utility feed, thereby transferring the load from the utility to the generators. This test is only performed when there are no surgeries scheduled and it is cancelled if there are any emergencies coming in all of the sudden. If the test is already in progress incoming ambulances with patients requiring surgery are diverted to another hospital. To finish the test the hospital is synchronized to the utility, the throttle on the generators turned down until no power is flowing through the generator breaker which is then opened. I’ve built and programmed control systems that do all this with multiple generators, multiple utility feeds, and multiple tiers of loads of different priorities.
        jimmygrapes 6y ago
        That is a good way to test the generator without interruption, but at some point you still need to test the ATS under real world circumstances: unstable or suddenly missing utility power. Throwing the main covers one of those, and the other requires a little more planning and resources.
        generatorguy 6y ago
        Honestly I am struggling to remember an instance where we purposefully plunged an operating hospital in to darkness. You can test the system’s ability to detect loss of power or poor quality power by lifting sensing wires or injecting out of frequency or voltage range power with a test set, and load pickup and load testing we used a load bank. The actual hospital load is fed through bypass breakers during these tests so if the utility went out we would have to manually operate breakers to disconnect the load bank and swing the vital loads over to the generators. If I am setting up electrical protection on a generator I don’t put a short circuit on the generator, I show the protection relay what it would see by injecting current and voltage with a relay test set and make sure the relay gives the signal to open the breaker when it should. Greenfield hospital generation sites are straightforward to open the utility feeds since the vital loads are still fed through the old power system.
jcrawfordor 6y ago
Some military equipment, especially naval, has something called a "battleshort" where electrical protections can be disabled during combat, where losing equipment could be more dangerous than electrical hazards. It's interesting to think about whether this could be applied to DC environments, but honestly, I think few DCs actually have this kind of risk aversion for downtime. Alternatives like precautionary replacement of generators that have been oversped probably come at too high of a cost, not to mention that many of those false-positive-prone protective devices are there to reduce risk of fire.
- captn3m0 6y ago
  Also on aircrafts: https://en.wikipedia.org/wiki/War_emergency_power https://en.wikipedia.org/wiki/War_emergency_power
  - irontinkerer 6y ago
    “ the P-51H Mustang was rated at 1,380 hp, but WEP would deliver up to 2,218 hp” Wow!
    - linsomniac 6y ago
      Indeed. I forget where I read it, but it provides that power increase for (IIRC) some small double digits number of seconds before you need to throttle back, or the engine eats itself. If you are able to make it back to base, and that WEP wire is broken, the engine needs a rebuild before it flies again. Definitely an impressive power boost though.
      - tomatotomato37 6y ago
        The MiG-25 also have something similar where the engine redline is marked around Mach 2.8, but there's no system stopping the pilot from keeping it maxed to somewhere above Mach 3 but damaging the engine in the process. Funnily enough I believe the SR-71 has the opposite problem, where the engines would merrily go above Mach 3.5 with no issue but the airframe and everything attached to it would get torn apart by the intense heat if the pilot tried to go any faster
eigenvector 6y ago
I work in the power generation industry. We have a lot of primary/secondary as well primary/standby systems. The cardinal rule is - test everything. Test it when you install and commission it, test it when part of the system changes, test it periodically during maintenance shutdowns, test it when you have a convenient time to do so without losing production (such as an outage to a different system that necessitates your system being offline). Test components individually - at the factory and in the field - and test systems end-to-end. Test with the most adverse possible conditions, not the most optimal. Test beyond your normal operating envelope. When I participate in design reviews as a maintenance engineer my primary line of inquiry is: how will I test this stuff during operations? Has it been designed in a way that makes testing impossible without an expensive outage? Can workers safely gain access to components that need routine testing without having to de-energize other, unrelated parts of the system? Now not every industry has the budget available to test to the extent that we do. But even in our industry, I often see a penny-wise, pound-foolish approach of "assume this thing works perfectly from the factory then act surprised years later when an abnormal condition occurs and it fails". If something was not tested, it may not work. It might be for a very simple reason (like mis-configuration of a single setting during installation) or a very complex reason. But either way, wouldn't the company rather know their equipment doesn't work -before- putting it into service than after?
- a1369209993 6y ago
  Apropos of this, in TFA's case they apparently never tested "rip the battery out while the system is running". (There are other, harder-to-test ways for a battery to fail, but that seems like a obvious and easily-tested failure mode.)
- kitteh 6y ago
  The telco Central Office I worked in had weekly tests where we'd run the generator for an hour. Everyone's desk computer required a UPS as they were on utility. Comically after a few years you'd have to get a new UPS at your desk as theyd die from continual use.
  - myself248 6y ago
copperx 6y ago
> But it's difficult to test this equipment You could just, you know, pull the plug and see what happens?
- CarbyAu 6y ago
  Yeah, but any enterprise that warrants this gear likely also has a bunch of risk averse bureaucracy saying "Unplug that, you lose your job."
  - bluGill 6y ago
    That is a mid level of risk adverse. High levels make you prove everything works by unplugging everything monthly.
    - yjftsjthsd-h 6y ago
      Risk aversion comes in shallow and deep variants.
mulmen 6y ago
My grade school neighbored a hospital. I recall occasional loud rumblings of the backup generators during recess, presumably they were tests. Legend was they came from a WWII submarine! Neat! Years later I had a tech support job in that hospital and all the PCs had small UPSs to keep them alive for the 30-60 seconds it took the big generators to come online. I never did get a chance to see the generator room. e: Another personal anecdote that is more relevant to the topic. I was working for a VOIP provider back when Hurricane Sandy hit NYC. Part of our production network was housed in a telecom building in Manhattan. We received daily updates on how many floors the flooding was from our equipment and about the fuel deliveries to the building backup generators. Apparently the generators were on the roof? I'd love to know more about the setup in that building because we had zero downtime.
- Fogest 6y ago
  Working in a 911 call center this is how it worked for us. Except we had larger UPS's that could last probably 30-60 minutes in the building and then a large external generator in a small building beside. You'd see stuff like the lights that weren't on the UPS's turn off for a few seconds before the main generator fired up.
  - linsomniac 6y ago
    Everything involved in 911 calls had special tagging to indicate it. Failure to go through appropriate review and approval processes before doing work on tagged equipment would create a Resume Generation Event.
  - mark-r 6y ago
    My current job refuses to replace old UPSs that have died, because they have a generator to keep the building powered in case of loss of power. But that doesn't keep your PC running for the moments it takes the generator to spin up.
    - Fogest 6y ago
      Oh yeah, if we had that where we worked people would definitely be in danger. Most likely all of our systems would be down if we didn't have the UPS+Generator setup. It often takes 5-30 seconds for the generator to kick in and be running at full capacity. Normally 5 seconds, but sometimes there is a bit of a delay.
cactus2093 6y ago
> it's difficult to test this equipment Can't you just flip the circuit breaker or just pull the plug out? Or does a real power outage usually look different to that in some way, like a surge followed by a shutoff?
- WalterBright 6y ago
  Around here the typical failure pattern is the power goes out for a second or two, then comes back on for a few seconds, then goes out for good.
  - myself248 6y ago
    That's a recloser operating. Initially, there's a fault, say, a tree contacting a wire. This conducts a bunch of current to ground, the voltage in part of the network sags a fair bit, and a protective relay senses the overcurrent and opens the circuit. This happens in a second or two, and your power goes out. You may notice lights dimming or flickering in the instant beforehand. But, many faults are transient. Perhaps that tree branch finished falling, and is no longer against the wire. So the protective relay performs a "reclose" operation, where it turns the power back on for a moment and measures the current. There's normally a huge starting inrush as motor-driven appliances restart, so it waits a moment before sampling the current, and this is the period when the power comes back on for a few seconds. (It's probably pretty nasty power during this interval, too.) If things are good, hey, you're good! A brief outage while waiting for the fault to clear itself, and then everything's back. The operation will be reported to the dispatch center, and someone may come out to inspect the area in case there's damage. But if the fault is still there, and after a moment the current is still way higher than it should be, the recloser opens again, and this time it stays open. Gonna need a crew to move that tree, and then they'll manually try again.
- zzz61831 6y ago
  You can put your equipment on a voltage relay to make failure modes as simple as flipping a circuit breaker. It will turn off once voltage drops too much and will wait for it to stabilize for specified amount of time to turn it on.

generatorguy 6y ago

Electrical Engineers would determine the settings for the electrical protection of a generator. Mechanical Engineers would determine the settings for mechanical protection of a motor. To operate the equipment beyond thresholds would result in reduced lifespan or immediate and permanent damage to the equipment. If this is what the customer requires they have to specify that to the Engineer so that when the equipment blows up the Engineer is not liable. Diesel generators for hospitals and water pumps for fire suppression systems are generally set up with very loose settings as it is clearly worse for a patient to die or a building to burn down than to destroy a machine, and this requirement is in the specifications.

brudgers 6y ago
Diesel generators for hospitals and water pumps for fire suppression systems Those systems also require regular testing by qualified personnel. NFPA 25 requires monthly testing of fire pumps and annual flow testing. For low rise buildings with inadequate municipal water pressure, a water tower might be cheaper and easier. NFPA 99 - Standard for health care facilities, requires emergency generators to be tested 12 times per year.
- generatorguy 6y ago
  The testing would consist of running the diesel generators under load but not operating in abnormal conditions.
  - bluGill 6y ago
    True, but the easiest test is to pull the main power to the hospital and watch the system automatically take over. If anything fails you know what is wrong. This is scheduled so that those in surgery (or other worrisome situations) have extra help and can signal get back to mains power now.
    - generatorguy 6y ago
      ideally the hospital’s generator control system or transfer switch is able to start and synchronize the generators and turn up the throttle until no power is being imported from the utility prior to opening the utility feed, thereby transferring the load from the utility to the generators. This test is only performed when there are no surgeries scheduled and it is cancelled if there are any emergencies coming in all of the sudden. If the test is already in progress incoming ambulances with patients requiring surgery are diverted to another hospital. To finish the test the hospital is synchronized to the utility, the throttle on the generators turned down until no power is flowing through the generator breaker which is then opened. I’ve built and programmed control systems that do all this with multiple generators, multiple utility feeds, and multiple tiers of loads of different priorities.
      - jimmygrapes 6y ago
        That is a good way to test the generator without interruption, but at some point you still need to test the ATS under real world circumstances: unstable or suddenly missing utility power. Throwing the main covers one of those, and the other requires a little more planning and resources.
        generatorguy 6y ago
        Honestly I am struggling to remember an instance where we purposefully plunged an operating hospital in to darkness. You can test the system’s ability to detect loss of power or poor quality power by lifting sensing wires or injecting out of frequency or voltage range power with a test set, and load pickup and load testing we used a load bank. The actual hospital load is fed through bypass breakers during these tests so if the utility went out we would have to manually operate breakers to disconnect the load bank and swing the vital loads over to the generators. If I am setting up electrical protection on a generator I don’t put a short circuit on the generator, I show the protection relay what it would see by injecting current and voltage with a relay test set and make sure the relay gives the signal to open the breaker when it should. Greenfield hospital generation sites are straightforward to open the utility feeds since the vital loads are still fed through the old power system.

jcrawfordor 6y ago

Some military equipment, especially naval, has something called a "battleshort" where electrical protections can be disabled during combat, where losing equipment could be more dangerous than electrical hazards. It's interesting to think about whether this could be applied to DC environments, but honestly, I think few DCs actually have this kind of risk aversion for downtime. Alternatives like precautionary replacement of generators that have been oversped probably come at too high of a cost, not to mention that many of those false-positive-prone protective devices are there to reduce risk of fire.

captn3m0 6y ago
Also on aircrafts: https://en.wikipedia.org/wiki/War_emergency_power https://en.wikipedia.org/wiki/War_emergency_power
- irontinkerer 6y ago
  “ the P-51H Mustang was rated at 1,380 hp, but WEP would deliver up to 2,218 hp” Wow!
  - linsomniac 6y ago
    Indeed. I forget where I read it, but it provides that power increase for (IIRC) some small double digits number of seconds before you need to throttle back, or the engine eats itself. If you are able to make it back to base, and that WEP wire is broken, the engine needs a rebuild before it flies again. Definitely an impressive power boost though.
    - tomatotomato37 6y ago
      The MiG-25 also have something similar where the engine redline is marked around Mach 2.8, but there's no system stopping the pilot from keeping it maxed to somewhere above Mach 3 but damaging the engine in the process. Funnily enough I believe the SR-71 has the opposite problem, where the engines would merrily go above Mach 3.5 with no issue but the airframe and everything attached to it would get torn apart by the intense heat if the pilot tried to go any faster

eigenvector 6y ago

I work in the power generation industry. We have a lot of primary/secondary as well primary/standby systems. The cardinal rule is - test everything. Test it when you install and commission it, test it when part of the system changes, test it periodically during maintenance shutdowns, test it when you have a convenient time to do so without losing production (such as an outage to a different system that necessitates your system being offline). Test components individually - at the factory and in the field - and test systems end-to-end. Test with the most adverse possible conditions, not the most optimal. Test beyond your normal operating envelope. When I participate in design reviews as a maintenance engineer my primary line of inquiry is: how will I test this stuff during operations? Has it been designed in a way that makes testing impossible without an expensive outage? Can workers safely gain access to components that need routine testing without having to de-energize other, unrelated parts of the system? Now not every industry has the budget available to test to the extent that we do. But even in our industry, I often see a penny-wise, pound-foolish approach of "assume this thing works perfectly from the factory then act surprised years later when an abnormal condition occurs and it fails". If something was not tested, it may not work. It might be for a very simple reason (like mis-configuration of a single setting during installation) or a very complex reason. But either way, wouldn't the company rather know their equipment doesn't work -before- putting it into service than after?

a1369209993 6y ago
Apropos of this, in TFA's case they apparently never tested "rip the battery out while the system is running". (There are other, harder-to-test ways for a battery to fail, but that seems like a obvious and easily-tested failure mode.)
kitteh 6y ago
The telco Central Office I worked in had weekly tests where we'd run the generator for an hour. Everyone's desk computer required a UPS as they were on utility. Comically after a few years you'd have to get a new UPS at your desk as theyd die from continual use.
- myself248 6y ago

copperx 6y ago

> But it's difficult to test this equipment You could just, you know, pull the plug and see what happens?

CarbyAu 6y ago
Yeah, but any enterprise that warrants this gear likely also has a bunch of risk averse bureaucracy saying "Unplug that, you lose your job."
- bluGill 6y ago
  That is a mid level of risk adverse. High levels make you prove everything works by unplugging everything monthly.
  - yjftsjthsd-h 6y ago
    Risk aversion comes in shallow and deep variants.

mulmen 6y ago

My grade school neighbored a hospital. I recall occasional loud rumblings of the backup generators during recess, presumably they were tests. Legend was they came from a WWII submarine! Neat! Years later I had a tech support job in that hospital and all the PCs had small UPSs to keep them alive for the 30-60 seconds it took the big generators to come online. I never did get a chance to see the generator room. e: Another personal anecdote that is more relevant to the topic. I was working for a VOIP provider back when Hurricane Sandy hit NYC. Part of our production network was housed in a telecom building in Manhattan. We received daily updates on how many floors the flooding was from our equipment and about the fuel deliveries to the building backup generators. Apparently the generators were on the roof? I'd love to know more about the setup in that building because we had zero downtime.

Fogest 6y ago
Working in a 911 call center this is how it worked for us. Except we had larger UPS's that could last probably 30-60 minutes in the building and then a large external generator in a small building beside. You'd see stuff like the lights that weren't on the UPS's turn off for a few seconds before the main generator fired up.
- linsomniac 6y ago
  Everything involved in 911 calls had special tagging to indicate it. Failure to go through appropriate review and approval processes before doing work on tagged equipment would create a Resume Generation Event.
- mark-r 6y ago
  My current job refuses to replace old UPSs that have died, because they have a generator to keep the building powered in case of loss of power. But that doesn't keep your PC running for the moments it takes the generator to spin up.
  - Fogest 6y ago
    Oh yeah, if we had that where we worked people would definitely be in danger. Most likely all of our systems would be down if we didn't have the UPS+Generator setup. It often takes 5-30 seconds for the generator to kick in and be running at full capacity. Normally 5 seconds, but sometimes there is a bit of a delay.

cactus2093 6y ago

> it's difficult to test this equipment Can't you just flip the circuit breaker or just pull the plug out? Or does a real power outage usually look different to that in some way, like a surge followed by a shutoff?

WalterBright 6y ago
Around here the typical failure pattern is the power goes out for a second or two, then comes back on for a few seconds, then goes out for good.
- myself248 6y ago
  That's a recloser operating. Initially, there's a fault, say, a tree contacting a wire. This conducts a bunch of current to ground, the voltage in part of the network sags a fair bit, and a protective relay senses the overcurrent and opens the circuit. This happens in a second or two, and your power goes out. You may notice lights dimming or flickering in the instant beforehand. But, many faults are transient. Perhaps that tree branch finished falling, and is no longer against the wire. So the protective relay performs a "reclose" operation, where it turns the power back on for a moment and measures the current. There's normally a huge starting inrush as motor-driven appliances restart, so it waits a moment before sampling the current, and this is the period when the power comes back on for a few seconds. (It's probably pretty nasty power during this interval, too.) If things are good, hey, you're good! A brief outage while waiting for the fault to clear itself, and then everything's back. The operation will be reported to the dispatch center, and someone may come out to inspect the area in case there's damage. But if the fault is still there, and after a moment the current is still way higher than it should be, the recloser opens again, and this time it stays open. Gonna need a crew to move that tree, and then they'll manually try again.
zzz61831 6y ago
You can put your equipment on a voltage relay to make failure modes as simple as flipping a circuit breaker. It will turn off once voltage drops too much and will wait for it to stabilize for specified amount of time to turn it on.