6 ms·

Ted Koppel (retired new anchor) began research for a book on this subject after retiring; the book, titled "Lights Out" was published late last year, i believe.

Ted Koppel (retired new anchor) began research for a book on this subject after retiring; the book, titled "Lights Out" was published late last year, i believe. Two things about this book were remarkable to me--one was the exhaustive research; the second was a remark from Koppel in the book (i don't recall the exact words he used nor where in the book it is) that it is difficult to imagine a better design for a Power Grid whose primary criterion was maximum exposure to catastrophic attack than the US Power Grid (used collectively to refer to three geographic grids).

phodo 10y ago
Thanks for the recommendation and insights on that book.
deleted 10y ago
[deleted]
pp19dd 10y ago
The critical piece of evidence presented in his book was the blackout of 2003. Ted Koppel's version was short and implied that the power grid system is so vulnerable that all it took was some overgrown trees to bring the northeast down to its knees for days. He then spent rest of the book talking about SCADA glitches and computing vulnerabilities and disaster preparations (focusing namely on the Mormon community, which is always prepared for a disaster.) Problem I had with the book is that he either misunderstood or misrepresented that triggering event. Yes, there were overgrown trees, high summer demand and temperatures that made power lines heat up and stretch. Wind speed dropped from 5 to 2 knots and cooled the line less than it normally did in that place. Cables stretched more than usual and grounded. These are all true facts. However, that's not what brought the power grid down that day. Here's what did. At 12:40 pm, an engineer working for MISO (transmission company, sort of like a traffic regulator) switched a regulation system to manual mode to make a correction. Note that the correction was made because the system had an errant piece of information regularly enough, which through a phone call the operator knew was false and corrected. The regulation system was called a "state estimator." It basically takes tons of factors, loads, schedules, downtimes, capacities, etc and optimizes power transmission. Like intelligent traffic control. Of course, all systems have input garbage from time to time, and he corrected this thing. At 1:30 PM that engineer went to lunch, forgetting to re-enable the automated mode. At 2:02 PM, a 345k-volt line tripped after touching a tree and started this cascading reaction resulting in a system blackout for a week. Other operators looking at the system readouts didn't know that the state estimator was now sending a false picture, being frozen in manual mode. By the time the unnamed engineer (from the NERC report) returned, it was too late. Other regulators, operators and power plants had been making uninformed decisions, preparations and adjustments for the power grid based on outdated, non-realtime information. However, the blackout didn't happen instantly -- it took four hours for a cascading chain reaction to trip every safety device in the system. Lets not even go into SCADA vulnerabilities. Here's some positive outcomes from the blackout: investigators made a staggering list of 46 recommendations -- none addressing tree trimming because vegetation events happened all the time. Those were routine. They did focus on computer security, even though there were no computer faults in this incident. Secondly, during the blackout the Department of Energy ordered state of Connecticut to energize one of their power plants despite their objections. They complied, and that restored power to metropolitan areas. This action is now a precedent for any future outage events: the legal framework for prioritizing power restoration exists and is in place. SCADA systems show up on HN often enough to be a point of ridicule. Good news is that the outdated SCADA systems are getting replaced with synchrophasors for the most critical power integration functions. Since 2003, nearly 1700 phasors were added to the national grid, and they're well on their way to be fitted in the roughly 7,000 power plants in the U.S. What it will do over SCADA is improve recovery times of an unstable power grid segment. Edit: derp, northeast, not nw. Also http://energy.gov/sites/prod/files/oeprod/DocumentsandMedia/BlackoutFinal-Web.pdf http://energy.gov/sites/prod/files/oeprod/DocumentsandMedia/...
- doug1001 10y ago
  needless to say, i lack the domain knowledge to critically evaluate the conclusions in Koppel's book, so always welcome to hear such evaluation from someone who does.
- wNk6A23YB 10y ago
  This post made me look up synchrophasors (cool name) and it turns out they rely on the GPS system for time synchronization and thus are susceptible to spoofing attacks[0]. It made me wonder if spoof-proof GPS via cryptography or similar was being working on. Apparently DHS was on that track in 2011 [1] but I'm not sure what came of it. There's more information in [2] about the risk of GPS spoofing and how utilities may be mitigating it. [0] http://gpsworld.com/wirelessinfrastructuregoing-against-time-13278/ http://gpsworld.com/wirelessinfrastructuregoing-against-time... [1]http://www.insidegnss.com/node/2824 http://www.insidegnss.com/node/2824 [2]http://www.eenews.net/stories/1060027181 http://www.eenews.net/stories/1060027181
- pjc50 10y ago
  The "synchrophasor" is interesting and caused me to think of the grid as a giant PLL: better detection of phase discrepancy allows the control system to improve the phase margin. (grid blackout == PLL loss of lock, too)
mark_l_watson 10y ago
I just checked Koppel's book out of the library and finished it. Great book and it pisses me off that we spend so much money on corporate welfare, human welfare, and wars overseas, but we can't spend much less money to harden the power grid. I read the the US government estimates a 80% civilian mortality if the power grid is down for a year.
- thingexplainer 10y ago
  > I read the the US government estimates a 80% civilian mortality if the power grid is down for a year. Our food production, waste and water treatment, hospitals, etc run on electricity, sure. But 80% doesn't seem anchored in reality. I feel like this irrationality about the risks posed by a blackout is what the article is meant to address.
  - mark_l_watson 10y ago
    I think a US Army report had the figure higher than 80%. A small thought experiment: imagine that you live in a city and for 1 year there was no electricity, no trash pickup, likely no water to drink and to flush toilets, and no gas for cars and trucks. I think this scenario is very unlikely, but it is still worthwhile hardening this infrastructure. I admit to being cautious. If it were up to me, I would also spend a lot of resources hardening computing systems: servers in business, kids' school laptops, home computers, IoT, network infrastructure, etc. would have maximum crypto, and we would have a "going to the moon" level effort pushing those aspects of computer science for general security issues.
  - JoeAltmaier 10y ago
    A year of blackout? Yeah, 80% sounds about right. See, our modern cities are designed around efficient transport (food, sanitation). Back in the day they added sewers to London because it took longer than a day for the nightsoil wagons to get to the dump sites and back - something more efficient was needed. Today its 1000X that. There's no way to get food to people without efficient transport, which depends utterly on energy. No way to get sewage out. No way to get water flowing. Heck even the traffic lights are needed to keep transport going. A year without food and water? Without sanitation? Of course nearly everybody dies. Manual solutions could be found, but its impossible that they could support the current population density. So folks would have to flee the cities. To the countryside, where they just die a different death. After pillaging and destroying any remaining food pipeline.

phodo 10y ago

Thanks for the recommendation and insights on that book.

deleted 10y ago

[deleted]

pp19dd 10y ago

The critical piece of evidence presented in his book was the blackout of 2003. Ted Koppel's version was short and implied that the power grid system is so vulnerable that all it took was some overgrown trees to bring the northeast down to its knees for days. He then spent rest of the book talking about SCADA glitches and computing vulnerabilities and disaster preparations (focusing namely on the Mormon community, which is always prepared for a disaster.) Problem I had with the book is that he either misunderstood or misrepresented that triggering event. Yes, there were overgrown trees, high summer demand and temperatures that made power lines heat up and stretch. Wind speed dropped from 5 to 2 knots and cooled the line less than it normally did in that place. Cables stretched more than usual and grounded. These are all true facts. However, that's not what brought the power grid down that day. Here's what did. At 12:40 pm, an engineer working for MISO (transmission company, sort of like a traffic regulator) switched a regulation system to manual mode to make a correction. Note that the correction was made because the system had an errant piece of information regularly enough, which through a phone call the operator knew was false and corrected. The regulation system was called a "state estimator." It basically takes tons of factors, loads, schedules, downtimes, capacities, etc and optimizes power transmission. Like intelligent traffic control. Of course, all systems have input garbage from time to time, and he corrected this thing. At 1:30 PM that engineer went to lunch, forgetting to re-enable the automated mode. At 2:02 PM, a 345k-volt line tripped after touching a tree and started this cascading reaction resulting in a system blackout for a week. Other operators looking at the system readouts didn't know that the state estimator was now sending a false picture, being frozen in manual mode. By the time the unnamed engineer (from the NERC report) returned, it was too late. Other regulators, operators and power plants had been making uninformed decisions, preparations and adjustments for the power grid based on outdated, non-realtime information. However, the blackout didn't happen instantly -- it took four hours for a cascading chain reaction to trip every safety device in the system. Lets not even go into SCADA vulnerabilities. Here's some positive outcomes from the blackout: investigators made a staggering list of 46 recommendations -- none addressing tree trimming because vegetation events happened all the time. Those were routine. They did focus on computer security, even though there were no computer faults in this incident. Secondly, during the blackout the Department of Energy ordered state of Connecticut to energize one of their power plants despite their objections. They complied, and that restored power to metropolitan areas. This action is now a precedent for any future outage events: the legal framework for prioritizing power restoration exists and is in place. SCADA systems show up on HN often enough to be a point of ridicule. Good news is that the outdated SCADA systems are getting replaced with synchrophasors for the most critical power integration functions. Since 2003, nearly 1700 phasors were added to the national grid, and they're well on their way to be fitted in the roughly 7,000 power plants in the U.S. What it will do over SCADA is improve recovery times of an unstable power grid segment. Edit: derp, northeast, not nw. Also http://energy.gov/sites/prod/files/oeprod/DocumentsandMedia/BlackoutFinal-Web.pdf http://energy.gov/sites/prod/files/oeprod/DocumentsandMedia/...

doug1001 10y ago
needless to say, i lack the domain knowledge to critically evaluate the conclusions in Koppel's book, so always welcome to hear such evaluation from someone who does.
wNk6A23YB 10y ago
This post made me look up synchrophasors (cool name) and it turns out they rely on the GPS system for time synchronization and thus are susceptible to spoofing attacks[0]. It made me wonder if spoof-proof GPS via cryptography or similar was being working on. Apparently DHS was on that track in 2011 [1] but I'm not sure what came of it. There's more information in [2] about the risk of GPS spoofing and how utilities may be mitigating it. [0] http://gpsworld.com/wirelessinfrastructuregoing-against-time-13278/ http://gpsworld.com/wirelessinfrastructuregoing-against-time... [1]http://www.insidegnss.com/node/2824 http://www.insidegnss.com/node/2824 [2]http://www.eenews.net/stories/1060027181 http://www.eenews.net/stories/1060027181
pjc50 10y ago
The "synchrophasor" is interesting and caused me to think of the grid as a giant PLL: better detection of phase discrepancy allows the control system to improve the phase margin. (grid blackout == PLL loss of lock, too)

mark_l_watson 10y ago

I just checked Koppel's book out of the library and finished it. Great book and it pisses me off that we spend so much money on corporate welfare, human welfare, and wars overseas, but we can't spend much less money to harden the power grid. I read the the US government estimates a 80% civilian mortality if the power grid is down for a year.

thingexplainer 10y ago
> I read the the US government estimates a 80% civilian mortality if the power grid is down for a year. Our food production, waste and water treatment, hospitals, etc run on electricity, sure. But 80% doesn't seem anchored in reality. I feel like this irrationality about the risks posed by a blackout is what the article is meant to address.
- mark_l_watson 10y ago
  I think a US Army report had the figure higher than 80%. A small thought experiment: imagine that you live in a city and for 1 year there was no electricity, no trash pickup, likely no water to drink and to flush toilets, and no gas for cars and trucks. I think this scenario is very unlikely, but it is still worthwhile hardening this infrastructure. I admit to being cautious. If it were up to me, I would also spend a lot of resources hardening computing systems: servers in business, kids' school laptops, home computers, IoT, network infrastructure, etc. would have maximum crypto, and we would have a "going to the moon" level effort pushing those aspects of computer science for general security issues.
- JoeAltmaier 10y ago
  A year of blackout? Yeah, 80% sounds about right. See, our modern cities are designed around efficient transport (food, sanitation). Back in the day they added sewers to London because it took longer than a day for the nightsoil wagons to get to the dump sites and back - something more efficient was needed. Today its 1000X that. There's no way to get food to people without efficient transport, which depends utterly on energy. No way to get sewage out. No way to get water flowing. Heck even the traffic lights are needed to keep transport going. A year without food and water? Without sanitation? Of course nearly everybody dies. Manual solutions could be found, but its impossible that they could support the current population density. So folks would have to flee the cities. To the countryside, where they just die a different death. After pillaging and destroying any remaining food pipeline.