5 ms·

> A black start would have required carefully rebooting a few power plants at a time and using them to jump-start others, thereby restoring the grid piece by pi

> A black start would have required carefully rebooting a few power plants at a time and using them to jump-start others, thereby restoring the grid piece by piece. It’s not a matter of flipping switches. The steps required for a black start are numerous, complex, and delicate. No one knows how long that process would take, because no one has ever needed to do it. Magness said it would have been weeks at least. I understand that it's a challenge to maintain 60-hertz frequency throughout the grid-restart process. But I'd like to know why exactly it'd take weeks instead of hours to restart the grid.

Filligree 5y ago
Because the procedure hasn't been tested. Nothing ever works right on the very first try.
c_o_n_v_e_x 5y ago
Cold starts on industrial equipment, turbo machinery, etc is not as simple as just turning the power back on. There are checklists and physical inspections that usually need to take place before re-instating. That's just on a single unit/plant, I have no idea how they coordinate across an entire power grid.
- mikeyouse 5y ago
  They basically can't. You run into the same problem that caused the blackout to begin with -- you need to turn on the generators one-by-one, but you also need to find an outlet for the power to exactly match the supply that's being generated. If you run a power plant without sufficient load, the frequency will flip in the other direction, and if you attach too much demand to a single generator that can't handle it, the frequency will drop again. So you need to do this slow 'titration' of turning on power assets, connecting demand to them, balancing that load and then moving on to the next thing. Texas has something like 11 designated 'black start' generators that are meant to be bulletproof but more than half of them failed in some way last February.
  - Tempest1981 5y ago
    Is this being done manually? Or under computer control? Seems like it could be automated.
    - YXNjaGVyZWdlbgo 5y ago
      You can't, it's just not feasible nor practical to have the required number of sensors for a cold start computer control on giant machinery.
    - JaimeThompson 5y ago
      If a lot of the power companies aren't willing to spend money on things like insulation and temperature sensors they are even less likely to spend what is necessary to automate a black start.
      - bluGill 5y ago
        I hope it is never worthwhile to automate a blackstart. Some parts of it maybe, but this should be a rare event that isn't worth the cost. I'm not sure how you would test your automation even if you tried.
    - p_l 5y ago
      In black start conditions, you probably aren't going to have reliable network or power to position the switches in startup position, or to reset breakers, possibly even to power the sensors necessary to bring back a single substation.
esalman 5y ago
You can take a look at this article [1], I think it does a good job of explaining why black-starting a power grid is risky and time consuming, especially when not maintained well. "It’s a large stepwise process to build up load, build up generation, build up more load, build up more generation until they’ve got enough reliability to go to the next element of the system..." There are added challenges in a distributed system. Black-starting a hydroelectric plant is easy but I doubt if Texas has any. [1] https://www.wired.com/story/venezuela-power-outage-black-start/ https://www.wired.com/story/venezuela-power-outage-black-sta...
- liketochill 5y ago
  That article is not wrong. It doesn’t go in to much technical detail. Black starting a plant is one thing which can basically be solved by having a working and large enough diesel generator on site to power the plant auxiliary systems such as oil and water pumps required for start up. Reenergizing all of the loads in a large system from zero is another thing. Energizing transmission lines with no load are essentially giant capacitors that initially appear as a dead short, energizing transformers that are essentially giant inductors that initially appear as a dead short, energizing resistive loads requires all of the sudden opening whatever the throttle on the power plant is - water flow, steam flow which itself requires gas/coal/feed water/etc in to a boiler, or natural gas, that is as rough on equipment as flooring the engine in your car except you have to make sure the power plants connected are as large as the loads being connected. And the inertia in the rotating masses in the system has to be enough to supply the energy to the load until those throttles open and generate the additional energy - time delays here are measured in seconds. How do the fish in the river or the people camping on the river bank like it when the river goes from 5% to 100% all of the sudden? More restrictions and considerations there. The whole process is a bunch of step responses to systems at unusual operating points. Operators of many facilities that normally just run at constant load points won’t be familiar with the unusual demands placed on the equipment. Equipment will be operating both at minimum and maximum limits in very short time spans. It’s a recipe for equipment failures unless it is practiced regularly. Solar and wind unless supplanted by batteries are no help here either.
  - 5y ago
reincarnate0x14 5y ago
The startup sequence first depends on manually switching circuits across the region because no power can be assumed, so trucks are rolling to hundreds of substations and generator switchyards. Then, coordinating with emergency phones and radios as available, manually switching in loads and self-startable generators into electrical islands in a careful balance to not re-trip everything again due to voltage collapse or frequency excursions. When a single generator synchronizes to a normally operating wide area grid, the relative power balance is so huge that if the generator is not close enough to matching the phasors it will get pulled into sync as long as it's close enough to not trip offline again to avoid damage. With only a few generators on line, and very narrow extra capacity and loads, the margin of error is much smaller. This must be done methodically and carefully for every generator in the black start plan, and it must be done with the majority of the region not having power so transportation, communication, possibly fuel services for the generators, and basic services like food and water and toilets are also impacted. Everything is going to be slow. Some plants also have ramp up and startup cycles that can take most of a day to produce close to their nameplate capacity, and most of them require a significant external power source to run. Coal plants require lots of moving machines that load and process and pulverize the coal and complex steam pressure control systems that don't go from 0 to 100 at a whim; Nuclear plant cooldowns after a trip can be days, and startup again may require another day or two as well. Modern combined cycle gas plants are relatively easy by comparison but still require several hundred KW of available station servicing external power to start the unit and usually take a few hours of careful checks to go from cold to ready to sync. It would probably only take a few days to get most of the ERCOT region back on line, but a full restoration would definitely take weeks as much of the work can't even begin until the big generators in key locations are all up and stable, and most of the efforts can't realistically proceed until the cause of the blackout has been resolved, e.g. no pumps for the gas lines, those generators can't start anyway.
themerone 5y ago
There is a lot of remote equipment that can't be remotely powered on or reconfigured after a full shutdown. Imagine a Y junction switch that can take switch between the power from 2 plants into a local city grid. The switch's control circuits might only having a power supply on the side of one plant. (Dumb cost saving measure, which I heard is falling out of practice.)

Filligree 5y ago

Because the procedure hasn't been tested. Nothing ever works right on the very first try.

c_o_n_v_e_x 5y ago

Cold starts on industrial equipment, turbo machinery, etc is not as simple as just turning the power back on. There are checklists and physical inspections that usually need to take place before re-instating. That's just on a single unit/plant, I have no idea how they coordinate across an entire power grid.

mikeyouse 5y ago
They basically can't. You run into the same problem that caused the blackout to begin with -- you need to turn on the generators one-by-one, but you also need to find an outlet for the power to exactly match the supply that's being generated. If you run a power plant without sufficient load, the frequency will flip in the other direction, and if you attach too much demand to a single generator that can't handle it, the frequency will drop again. So you need to do this slow 'titration' of turning on power assets, connecting demand to them, balancing that load and then moving on to the next thing. Texas has something like 11 designated 'black start' generators that are meant to be bulletproof but more than half of them failed in some way last February.
- Tempest1981 5y ago
  Is this being done manually? Or under computer control? Seems like it could be automated.
  - YXNjaGVyZWdlbgo 5y ago
    You can't, it's just not feasible nor practical to have the required number of sensors for a cold start computer control on giant machinery.
  - JaimeThompson 5y ago
    If a lot of the power companies aren't willing to spend money on things like insulation and temperature sensors they are even less likely to spend what is necessary to automate a black start.
    - bluGill 5y ago
      I hope it is never worthwhile to automate a blackstart. Some parts of it maybe, but this should be a rare event that isn't worth the cost. I'm not sure how you would test your automation even if you tried.
  - p_l 5y ago
    In black start conditions, you probably aren't going to have reliable network or power to position the switches in startup position, or to reset breakers, possibly even to power the sensors necessary to bring back a single substation.

esalman 5y ago

You can take a look at this article [1], I think it does a good job of explaining why black-starting a power grid is risky and time consuming, especially when not maintained well. "It’s a large stepwise process to build up load, build up generation, build up more load, build up more generation until they’ve got enough reliability to go to the next element of the system..." There are added challenges in a distributed system. Black-starting a hydroelectric plant is easy but I doubt if Texas has any. [1] https://www.wired.com/story/venezuela-power-outage-black-start/ https://www.wired.com/story/venezuela-power-outage-black-sta...

liketochill 5y ago
That article is not wrong. It doesn’t go in to much technical detail. Black starting a plant is one thing which can basically be solved by having a working and large enough diesel generator on site to power the plant auxiliary systems such as oil and water pumps required for start up. Reenergizing all of the loads in a large system from zero is another thing. Energizing transmission lines with no load are essentially giant capacitors that initially appear as a dead short, energizing transformers that are essentially giant inductors that initially appear as a dead short, energizing resistive loads requires all of the sudden opening whatever the throttle on the power plant is - water flow, steam flow which itself requires gas/coal/feed water/etc in to a boiler, or natural gas, that is as rough on equipment as flooring the engine in your car except you have to make sure the power plants connected are as large as the loads being connected. And the inertia in the rotating masses in the system has to be enough to supply the energy to the load until those throttles open and generate the additional energy - time delays here are measured in seconds. How do the fish in the river or the people camping on the river bank like it when the river goes from 5% to 100% all of the sudden? More restrictions and considerations there. The whole process is a bunch of step responses to systems at unusual operating points. Operators of many facilities that normally just run at constant load points won’t be familiar with the unusual demands placed on the equipment. Equipment will be operating both at minimum and maximum limits in very short time spans. It’s a recipe for equipment failures unless it is practiced regularly. Solar and wind unless supplanted by batteries are no help here either.
- 5y ago

reincarnate0x14 5y ago

The startup sequence first depends on manually switching circuits across the region because no power can be assumed, so trucks are rolling to hundreds of substations and generator switchyards. Then, coordinating with emergency phones and radios as available, manually switching in loads and self-startable generators into electrical islands in a careful balance to not re-trip everything again due to voltage collapse or frequency excursions. When a single generator synchronizes to a normally operating wide area grid, the relative power balance is so huge that if the generator is not close enough to matching the phasors it will get pulled into sync as long as it's close enough to not trip offline again to avoid damage. With only a few generators on line, and very narrow extra capacity and loads, the margin of error is much smaller. This must be done methodically and carefully for every generator in the black start plan, and it must be done with the majority of the region not having power so transportation, communication, possibly fuel services for the generators, and basic services like food and water and toilets are also impacted. Everything is going to be slow. Some plants also have ramp up and startup cycles that can take most of a day to produce close to their nameplate capacity, and most of them require a significant external power source to run. Coal plants require lots of moving machines that load and process and pulverize the coal and complex steam pressure control systems that don't go from 0 to 100 at a whim; Nuclear plant cooldowns after a trip can be days, and startup again may require another day or two as well. Modern combined cycle gas plants are relatively easy by comparison but still require several hundred KW of available station servicing external power to start the unit and usually take a few hours of careful checks to go from cold to ready to sync. It would probably only take a few days to get most of the ERCOT region back on line, but a full restoration would definitely take weeks as much of the work can't even begin until the big generators in key locations are all up and stable, and most of the efforts can't realistically proceed until the cause of the blackout has been resolved, e.g. no pumps for the gas lines, those generators can't start anyway.

themerone 5y ago

There is a lot of remote equipment that can't be remotely powered on or reconfigured after a full shutdown. Imagine a Y junction switch that can take switch between the power from 2 plants into a local city grid. The switch's control circuits might only having a power supply on the side of one plant. (Dumb cost saving measure, which I heard is falling out of practice.)