8 ms·

The Threat to Americas Electrical Grid Is Bigger Than Imagined

91 points by weatherlight 10y ago

vibrio 10y ago
Today I learned that my dog is doing her part to improve National Security.
danielvf 10y ago
The US has actually has three independent power grids: East, West, and unsurprisingly, Texas.
- vibrio 10y ago
  Some would say the US has two grids and Texas had one.
- nayuki 10y ago
  This is coincidentally similar to the situation in Canada with Quebec on their own power grid
doug1001 10y ago
Ted Koppel (retired new anchor) began research for a book on this subject after retiring; the book, titled "Lights Out" was published late last year, i believe. Two things about this book were remarkable to me--one was the exhaustive research; the second was a remark from Koppel in the book (i don't recall the exact words he used nor where in the book it is) that it is difficult to imagine a better design for a Power Grid whose primary criterion was maximum exposure to catastrophic attack than the US Power Grid (used collectively to refer to three geographic grids).
- phodo 10y ago
  Thanks for the recommendation and insights on that book.
- deleted 10y ago
  [deleted]
- pp19dd 10y ago
  The critical piece of evidence presented in his book was the blackout of 2003. Ted Koppel's version was short and implied that the power grid system is so vulnerable that all it took was some overgrown trees to bring the northeast down to its knees for days. He then spent rest of the book talking about SCADA glitches and computing vulnerabilities and disaster preparations (focusing namely on the Mormon community, which is always prepared for a disaster.) Problem I had with the book is that he either misunderstood or misrepresented that triggering event. Yes, there were overgrown trees, high summer demand and temperatures that made power lines heat up and stretch. Wind speed dropped from 5 to 2 knots and cooled the line less than it normally did in that place. Cables stretched more than usual and grounded. These are all true facts. However, that's not what brought the power grid down that day. Here's what did. At 12:40 pm, an engineer working for MISO (transmission company, sort of like a traffic regulator) switched a regulation system to manual mode to make a correction. Note that the correction was made because the system had an errant piece of information regularly enough, which through a phone call the operator knew was false and corrected. The regulation system was called a "state estimator." It basically takes tons of factors, loads, schedules, downtimes, capacities, etc and optimizes power transmission. Like intelligent traffic control. Of course, all systems have input garbage from time to time, and he corrected this thing. At 1:30 PM that engineer went to lunch, forgetting to re-enable the automated mode. At 2:02 PM, a 345k-volt line tripped after touching a tree and started this cascading reaction resulting in a system blackout for a week. Other operators looking at the system readouts didn't know that the state estimator was now sending a false picture, being frozen in manual mode. By the time the unnamed engineer (from the NERC report) returned, it was too late. Other regulators, operators and power plants had been making uninformed decisions, preparations and adjustments for the power grid based on outdated, non-realtime information. However, the blackout didn't happen instantly -- it took four hours for a cascading chain reaction to trip every safety device in the system. Lets not even go into SCADA vulnerabilities. Here's some positive outcomes from the blackout: investigators made a staggering list of 46 recommendations -- none addressing tree trimming because vegetation events happened all the time. Those were routine. They did focus on computer security, even though there were no computer faults in this incident. Secondly, during the blackout the Department of Energy ordered state of Connecticut to energize one of their power plants despite their objections. They complied, and that restored power to metropolitan areas. This action is now a precedent for any future outage events: the legal framework for prioritizing power restoration exists and is in place. SCADA systems show up on HN often enough to be a point of ridicule. Good news is that the outdated SCADA systems are getting replaced with synchrophasors for the most critical power integration functions. Since 2003, nearly 1700 phasors were added to the national grid, and they're well on their way to be fitted in the roughly 7,000 power plants in the U.S. What it will do over SCADA is improve recovery times of an unstable power grid segment. Edit: derp, northeast, not nw. Also http://energy.gov/sites/prod/files/oeprod/DocumentsandMedia/BlackoutFinal-Web.pdf http://energy.gov/sites/prod/files/oeprod/DocumentsandMedia/...
- mark_l_watson 10y ago
  I just checked Koppel's book out of the library and finished it. Great book and it pisses me off that we spend so much money on corporate welfare, human welfare, and wars overseas, but we can't spend much less money to harden the power grid. I read the the US government estimates a 80% civilian mortality if the power grid is down for a year.
  - thingexplainer 10y ago
    > I read the the US government estimates a 80% civilian mortality if the power grid is down for a year. Our food production, waste and water treatment, hospitals, etc run on electricity, sure. But 80% doesn't seem anchored in reality. I feel like this irrationality about the risks posed by a blackout is what the article is meant to address.
    - mark_l_watson 10y ago
      I think a US Army report had the figure higher than 80%. A small thought experiment: imagine that you live in a city and for 1 year there was no electricity, no trash pickup, likely no water to drink and to flush toilets, and no gas for cars and trucks. I think this scenario is very unlikely, but it is still worthwhile hardening this infrastructure. I admit to being cautious. If it were up to me, I would also spend a lot of resources hardening computing systems: servers in business, kids' school laptops, home computers, IoT, network infrastructure, etc. would have maximum crypto, and we would have a "going to the moon" level effort pushing those aspects of computer science for general security issues.
    - JoeAltmaier 10y ago
      A year of blackout? Yeah, 80% sounds about right. See, our modern cities are designed around efficient transport (food, sanitation). Back in the day they added sewers to London because it took longer than a day for the nightsoil wagons to get to the dump sites and back - something more efficient was needed. Today its 1000X that. There's no way to get food to people without efficient transport, which depends utterly on energy. No way to get sewage out. No way to get water flowing. Heck even the traffic lights are needed to keep transport going. A year without food and water? Without sanitation? Of course nearly everybody dies. Manual solutions could be found, but its impossible that they could support the current population density. So folks would have to flee the cities. To the countryside, where they just die a different death. After pillaging and destroying any remaining food pipeline.
munin 10y ago
Everything that has an off switch has an on switch. Attacks against the power grid are weak because, in almost every scenario you can conjure, the remediation is just disconnecting the network and flipping the on switch. The things to look for are asymmetries. What is something that is very easy to do but very hard to un-do? What is a very important thing that would impact lots of people that is very easy to do or set in motion, but much harder to un-do or stop? Look for those and you'll find actual problems. There are far fewer of them and they are difficult (requiring creativity and domain expertise) to find, which is why no one talks about them...
- TeMPOraL 10y ago
  > Everything that has an off switch has an on switch. Not really. The ultimate off switch for every appliance is that appliance burning out. And quite a lot of dangers to the power grid include, or would cause, physical destruction of key infrastructure elements.
  - munin 10y ago
    that scenario is covered in the second paragraph ;)
- tlb 10y ago
  Yes. The biggest risk is of process controllers being reconfigured to cause physical damage. For instance, gas turbines can be cranked up to speeds & temperatures at which the turbine blades disintegrate. Or you can shut off combustion but keep fuel flowing, then restart combustion and get an explosion in the exhaust system. Why not just design equipment to have analog fail-safes? To get the last few percent of efficiency out of many systems you have to push them beyond what can be simply protected with fuses and analog shutoffs, and depend on software. Nuclear reactors can't just be turned off. You have to keep the cooling pumps running or the core overheats. Solar thermal farms can be hacked to point all the mirrors wrong and melt the tower. Large windmills have very complicated wind speed/blade angle controls, and can be made to disintegrate in minutes with the right inputs. All the effects I mentioned are local to one system. It's not clear if there's an attack that depends on the dynamics of the overall grid. But I can understand why researchers focus on the largest possible abstraction, rather than writing detailed engineering papers about turbine failure modes.
nxzero 10y ago
Key lesson is that if you're the Russians, train squirrels to attack the power grid, deploy them, and then deny, deny, deny.
- ilaksh 10y ago
  In all seriousness, most of the squirrels are already working for the Russians.
Animats 10y ago
With all the money we spend on "homeland security", there really should be a reserve supply of large transformers, just in case.
- idlewords 10y ago
  Ideally Autobots.
- shorttime 10y ago
  A lot of power plants do, at least the ones I've worked at. I wouldn't be surprised if these substations don't. Inspection teams come in all the time and practically force companies to do things like obtaining a spare.
msisk6 10y ago
I work for a grid operator here in the US. There's certainly a lot of FUD in that article, but it gets one thing right: critters in the equipment are a major source of failure. Last year we had a power glitch at our office when a snake climbed up a power pole and shorted across two phases. The snake didn't survive and folks learned to make sure their desktops are plugged into the outlets served by the building UPS.

91 points by weatherlight 10y ago

vibrio 10y ago

Today I learned that my dog is doing her part to improve National Security.

danielvf 10y ago

The US has actually has three independent power grids: East, West, and unsurprisingly, Texas.

vibrio 10y ago
Some would say the US has two grids and Texas had one.
nayuki 10y ago
This is coincidentally similar to the situation in Canada with Quebec on their own power grid

doug1001 10y ago

Ted Koppel (retired new anchor) began research for a book on this subject after retiring; the book, titled "Lights Out" was published late last year, i believe. Two things about this book were remarkable to me--one was the exhaustive research; the second was a remark from Koppel in the book (i don't recall the exact words he used nor where in the book it is) that it is difficult to imagine a better design for a Power Grid whose primary criterion was maximum exposure to catastrophic attack than the US Power Grid (used collectively to refer to three geographic grids).

phodo 10y ago
Thanks for the recommendation and insights on that book.
deleted 10y ago
[deleted]
pp19dd 10y ago
The critical piece of evidence presented in his book was the blackout of 2003. Ted Koppel's version was short and implied that the power grid system is so vulnerable that all it took was some overgrown trees to bring the northeast down to its knees for days. He then spent rest of the book talking about SCADA glitches and computing vulnerabilities and disaster preparations (focusing namely on the Mormon community, which is always prepared for a disaster.) Problem I had with the book is that he either misunderstood or misrepresented that triggering event. Yes, there were overgrown trees, high summer demand and temperatures that made power lines heat up and stretch. Wind speed dropped from 5 to 2 knots and cooled the line less than it normally did in that place. Cables stretched more than usual and grounded. These are all true facts. However, that's not what brought the power grid down that day. Here's what did. At 12:40 pm, an engineer working for MISO (transmission company, sort of like a traffic regulator) switched a regulation system to manual mode to make a correction. Note that the correction was made because the system had an errant piece of information regularly enough, which through a phone call the operator knew was false and corrected. The regulation system was called a "state estimator." It basically takes tons of factors, loads, schedules, downtimes, capacities, etc and optimizes power transmission. Like intelligent traffic control. Of course, all systems have input garbage from time to time, and he corrected this thing. At 1:30 PM that engineer went to lunch, forgetting to re-enable the automated mode. At 2:02 PM, a 345k-volt line tripped after touching a tree and started this cascading reaction resulting in a system blackout for a week. Other operators looking at the system readouts didn't know that the state estimator was now sending a false picture, being frozen in manual mode. By the time the unnamed engineer (from the NERC report) returned, it was too late. Other regulators, operators and power plants had been making uninformed decisions, preparations and adjustments for the power grid based on outdated, non-realtime information. However, the blackout didn't happen instantly -- it took four hours for a cascading chain reaction to trip every safety device in the system. Lets not even go into SCADA vulnerabilities. Here's some positive outcomes from the blackout: investigators made a staggering list of 46 recommendations -- none addressing tree trimming because vegetation events happened all the time. Those were routine. They did focus on computer security, even though there were no computer faults in this incident. Secondly, during the blackout the Department of Energy ordered state of Connecticut to energize one of their power plants despite their objections. They complied, and that restored power to metropolitan areas. This action is now a precedent for any future outage events: the legal framework for prioritizing power restoration exists and is in place. SCADA systems show up on HN often enough to be a point of ridicule. Good news is that the outdated SCADA systems are getting replaced with synchrophasors for the most critical power integration functions. Since 2003, nearly 1700 phasors were added to the national grid, and they're well on their way to be fitted in the roughly 7,000 power plants in the U.S. What it will do over SCADA is improve recovery times of an unstable power grid segment. Edit: derp, northeast, not nw. Also http://energy.gov/sites/prod/files/oeprod/DocumentsandMedia/BlackoutFinal-Web.pdf http://energy.gov/sites/prod/files/oeprod/DocumentsandMedia/...
mark_l_watson 10y ago
I just checked Koppel's book out of the library and finished it. Great book and it pisses me off that we spend so much money on corporate welfare, human welfare, and wars overseas, but we can't spend much less money to harden the power grid. I read the the US government estimates a 80% civilian mortality if the power grid is down for a year.
- thingexplainer 10y ago
  > I read the the US government estimates a 80% civilian mortality if the power grid is down for a year. Our food production, waste and water treatment, hospitals, etc run on electricity, sure. But 80% doesn't seem anchored in reality. I feel like this irrationality about the risks posed by a blackout is what the article is meant to address.
  - mark_l_watson 10y ago
    I think a US Army report had the figure higher than 80%. A small thought experiment: imagine that you live in a city and for 1 year there was no electricity, no trash pickup, likely no water to drink and to flush toilets, and no gas for cars and trucks. I think this scenario is very unlikely, but it is still worthwhile hardening this infrastructure. I admit to being cautious. If it were up to me, I would also spend a lot of resources hardening computing systems: servers in business, kids' school laptops, home computers, IoT, network infrastructure, etc. would have maximum crypto, and we would have a "going to the moon" level effort pushing those aspects of computer science for general security issues.
  - JoeAltmaier 10y ago
    A year of blackout? Yeah, 80% sounds about right. See, our modern cities are designed around efficient transport (food, sanitation). Back in the day they added sewers to London because it took longer than a day for the nightsoil wagons to get to the dump sites and back - something more efficient was needed. Today its 1000X that. There's no way to get food to people without efficient transport, which depends utterly on energy. No way to get sewage out. No way to get water flowing. Heck even the traffic lights are needed to keep transport going. A year without food and water? Without sanitation? Of course nearly everybody dies. Manual solutions could be found, but its impossible that they could support the current population density. So folks would have to flee the cities. To the countryside, where they just die a different death. After pillaging and destroying any remaining food pipeline.

munin 10y ago

Everything that has an off switch has an on switch. Attacks against the power grid are weak because, in almost every scenario you can conjure, the remediation is just disconnecting the network and flipping the on switch. The things to look for are asymmetries. What is something that is very easy to do but very hard to un-do? What is a very important thing that would impact lots of people that is very easy to do or set in motion, but much harder to un-do or stop? Look for those and you'll find actual problems. There are far fewer of them and they are difficult (requiring creativity and domain expertise) to find, which is why no one talks about them...

TeMPOraL 10y ago
> Everything that has an off switch has an on switch. Not really. The ultimate off switch for every appliance is that appliance burning out. And quite a lot of dangers to the power grid include, or would cause, physical destruction of key infrastructure elements.
- munin 10y ago
  that scenario is covered in the second paragraph ;)
tlb 10y ago
Yes. The biggest risk is of process controllers being reconfigured to cause physical damage. For instance, gas turbines can be cranked up to speeds & temperatures at which the turbine blades disintegrate. Or you can shut off combustion but keep fuel flowing, then restart combustion and get an explosion in the exhaust system. Why not just design equipment to have analog fail-safes? To get the last few percent of efficiency out of many systems you have to push them beyond what can be simply protected with fuses and analog shutoffs, and depend on software. Nuclear reactors can't just be turned off. You have to keep the cooling pumps running or the core overheats. Solar thermal farms can be hacked to point all the mirrors wrong and melt the tower. Large windmills have very complicated wind speed/blade angle controls, and can be made to disintegrate in minutes with the right inputs. All the effects I mentioned are local to one system. It's not clear if there's an attack that depends on the dynamics of the overall grid. But I can understand why researchers focus on the largest possible abstraction, rather than writing detailed engineering papers about turbine failure modes.

nxzero 10y ago

Key lesson is that if you're the Russians, train squirrels to attack the power grid, deploy them, and then deny, deny, deny.

ilaksh 10y ago
In all seriousness, most of the squirrels are already working for the Russians.

Animats 10y ago

With all the money we spend on "homeland security", there really should be a reserve supply of large transformers, just in case.

idlewords 10y ago
Ideally Autobots.
shorttime 10y ago
A lot of power plants do, at least the ones I've worked at. I wouldn't be surprised if these substations don't. Inspection teams come in all the time and practically force companies to do things like obtaining a spare.

msisk6 10y ago

I work for a grid operator here in the US. There's certainly a lot of FUD in that article, but it gets one thing right: critters in the equipment are a major source of failure. Last year we had a power glitch at our office when a snake climbed up a power pole and shorted across two phases. The snake didn't survive and folks learned to make sure their desktops are plugged into the outlets served by the building UPS.