46 ms·

US unprepared for an electrical grid collapse, but it costs only $300M

303 points by rictic 6y ago

csecdaemon 6y ago
While I don't disagree that transformers... let alone key transformer.. are critical. By disabling 9 transformers you cannot take down “THE GRID”. You could certainly cause havoc and perhaps take down sections of a grid, perhaps a large one such as new york city. But these would be localized. While ensuring that critical components are local and available is important much of this is being done in the energy sector. It’s not to say there simply are no replacements. However, the energy sector, including transmission, is made up of over 6000 separate organizations. Most of which are not federally controlled or controlled by the local government. Put simply they are for-profit companies. They are critical infrastructure and regulated by the gov, but not a part of. Put short, this is a serious matter, but not in the way it is presented here. It would be a herculean effort to take out 1 let alone the suggested 9. Especially in a coordinated fashion. EVEN IF it was achieved.. This would only cause localized issues. Not takedown. “THE GRID” This is a much larger discussion but a few things to consider: 1. There is no singular "grid". It just does not work that way. There are thousands of generation and transmission companies throughout the US. They all maintain there own “grids”. There is connectivity to provide ways to connect grids to sell/buy/shed power. But these connections are controlled and can be simply… disconnected. 2. Without going on a major diatribe on the many different attack vectors and inherent vulnerabilities in the energy sector…. Suffice to say.. The easiest and most sure way to take out a transformer is physical. They would need to be destroyed or disabled. Digital/remote interference would require an immense effort and campaign… we are not talking breaching the 9 locations… we’re talking hacking millions upon millions of connected devices and coordinating load to take out a transformer or the grid as whole…. The coordination and effort necessary for either are staggering. 3. There are in fact spares and inventory… just not in the hands of the feds.
- tonyedgecombe 6y ago
  You seem to be assuming some sort of terrorist event. I would have thought the most likely reason for the whole grid to go down is another Carrington event.
  - csecdaemon 6y ago
    I'm not assuming anything/anyone specifically but lean towards nation-state, anarchist, terror, etc. A natural disaster such as a flare, earthquake, fire, etc. are a different story. The radiation and electrical magnetic pulse from a solar flare could hamper equipment. However recent studies in the last few years by EPRI, INL, and ORNL actually show that much equipment would survive and equipment that did not would be localized.
    - FlyMoreRockets 6y ago
      Care to link to any of those studies? I'd love to read them.
      - csecdaemon 6y ago
        Here is EPRI report - https://www.epri.com/#/pages/product/3002014979/?lang=en-US https://www.epri.com/#/pages/product/3002014979/?lang=en-US
    - runawaybottle 6y ago
      Do we stress test our grid(s) core components with inputs that would be similar to that of a solar flare to write any of this off? I feel like 5 months ago we’d be sitting here dismissing the notion that a face mask shortage is a trivial problem, all countries can just suddenly ramp up production. But we couldn’t.
- FpUser 6y ago
  There is no singular "grid". It just does not work that way This did not prevent "Northeast blackout of 2003" when bunch of states in the US and province of Ontario in Canada went without power for 3 days. I remember it quite clearly. Was not big fun. Luckily my friends and I with families went camping on the lake so did not really suffer other than some little disaster in a fridge ;)
  - runawaybottle 6y ago
    From wiki The blackout's proximate cause was a software bug in the alarm system at the control room of FirstEnergy, an Akron, Ohio–based company, which rendered operators unaware of the need to redistribute load after overloaded transmission lines drooped into foliage. What should have been a manageable local blackout cascaded into the collapse of the entire Northeast region. How fragile we are.
  - makomk 6y ago
    That was a cascading failure where the instability of part of the grid tripped protection systems intended to prevent damage to equipment throughout the rest. The individual regions making up that grid were perfectly capable of operating independently, possibly with reduced capacity, even after the blackout. What they couldn't do is remain operational whilst unexpectedly being cut off, because that caused transient unbalances between generation and load. I'm pretty sure there's no failure of a small number of transformers that would leave the US as a whole, or any large chunk of it, without power for a prolonged period. There are probably scenarios where it could trigger a cascading failure where protection devices shut down the rest of the grid in order to protect the other equipment from imbalances caused by the failures, but that alone is not going to lead to 12-18 months of blackouts and the death of 90% of the US population.
xkapastel 6y ago
> In testimony before a Congressional Committee, it has been asserted that a prolonged collapse of this nation’s electrical grid—through starvation, disease, and societal collapse—could result in the death of up to 90% of the American population. This could happen at any time due to a solar flare. The coronavirus has made it pretty clear that America is completely unprepared for any sort of major deviation from "normal".
- deleted 6y ago
  [deleted]
- intopieces 6y ago
  > The coronavirus has made it pretty clear that America is completely unprepared for any sort of major deviation from "normal". This is the nature of the US system of competitive federalism. States only coordinate with the federal government when either (a) it's politically expedient, because the ruling party in the same for both at that time, or (b) the federal government forces this issue, at which time lawsuits ensue. It's helpful to think of the US as more a series of regions that sometimes fall under a shifting umbrella government than an actual, single entity.
  - magicsmoke 6y ago
    It's pretty interesting that most countries in Europe are unitary states while most countries in the Americas are federal states. Maybe due to the centuries of warfare in Europe the states that survived were unitary states that could better handle external threats. Or maybe it's a holdover from a tradition of monarchy that the Americas never had. Interestingly, Germany is also a federal state. Yet we never seem to hear about federal regions of Germany disagreeing with each other and fighting with the Federal government like we hear about the US. Any Germans want to spill the tea on German inter-regional political drama?
    - runawaybottle 6y ago
      It can be argued that this it is part of the American political character because our origin story was shaped by the combination of disparate states, as in, we could have easily had colonies that sided with the British. It got cemented due to Slavery as we basically carved up the country between slaves states and free states, and ultimately fought a civil war to deal with this bifurcation. It’s been our long history, it defined us. Edit: Whoops, I did the American thing again and forgot there are other countries in the Americas.
PacketPaul 6y ago
The book “One Second After” deals with an EMP attack on the US. I don’t understand why we are not better prepared.
- barkingcat 6y ago
  because it costs money and decreases profits.
- madengr 6y ago
  We are, thats what the military and intelligence agencies do, and why we have a retaliatory nuclear arsenal.
wcoenen 6y ago
> I’d like to ask you guys whether we are prepared for other black swans. Let’s start with a collapse of the electrical system due to solar flares or electromagnetic pulse attacks. A Black Swan is a "high-profile, hard-to-predict, and rare event that is beyond the realm of normal expectations". But we know that the Carrington Event happened in 1859, and that it's only a matter of time before a big solar flare will affect Earth in the same way. So this is not a Black Swan. But yes, we should prepare for foreseeable rare events.
- kiba 6y ago
  A Black Swan is probably more about the POV of an actor as opposed to somebody who knows this, then it's not a black swan.
- beambot 6y ago
  Events like the Carrington & Covid seem to fit: high profile, hard to predict and rare. IMO, both qualify as Black swan. At some point, you have to pick a threshold on the probabilities involved... These events are likely independent & identically distributed (i.e. hard to predict for any year), and I'd call sub-1% annual chance pretty rare. But I'm receptive to a different threshold... But you can't just say "Black swan only counts for previously-inconceivable events". Edit: I'm assuming covid-19 is structurally different than Sars, mers, Ebola, & H1N1 purely based on the global effect it has had to date. If you found those others as pandemics (not according to WHO though), then you can remove it as Black swan.
  - AQuantized 6y ago
    I think they are both categorically grey swans. https://www.investopedia.com/terms/g/grey-swan.asp https://www.investopedia.com/terms/g/grey-swan.asp COVID-19 especially because of the previous impact of Spanish Flu.
  - Izkata 6y ago
    > These events are likely independent & identically distributed (i.e. hard to predict for any year), and I'd call sub-1% annual chance pretty rare. Put another way, it seems reasonable to me that "no one alive has experienced it before (and probably not their parents either)" is a decent threshold.
renewiltord 6y ago
Safrr Arkstorm or Tsunami are also pretty cheap to take care of. But that's life.
flipgimble 6y ago
This article has an unexpected resonance for me after several weeks of pandemic lockdown. Just three months ago I would have laughed it off. Now I am increasingly pissed off. In the US, but I see reflection of this in Europe as well, has allowed irresponsible puerile imbeciles to be elected. These people play games with wealth and power, with no regard to the lives and well-being of the society that pays for it. I did not mention any party, that perennial misdirection to their collective heist. Also don't fall for the other smoke screen of small vs. big government. I demand at least a competent government. For politicians truth is whatever you make people believe it to be. Meanwhile engineers and scientists struggle daily to find the hard facts of the world to painstakingly build or discover something that improves lives. We should be disgusted and revolted at who is leading us right now. We should drive these fools out office and strip them of any respect. If we don't, there will be no one competent left in position to rebuild from this, or the next, catastrophe that "nobody" could have expected. Or maybe there will be some states or localities that can get their act together, and the rest will be a wasteland of warring cultural and racial tribes.
- twomoretime 6y ago
  There's an old saying that people deserve their rulers.
  - Afforess 6y ago
    This attitude is toxic and worse, incorrect. The majority of Americans did not vote for our President. I live in Austin, and my state district in TX includes a tiny slice of a city hundreds of miles away, Houston. Many state districts are like this, gerrymandered to prevent my vote from counting. Some places have it worse, DC, the capital, has no representation at all (fun trivia: Google the DC license plate). The voters didn't ask nor vote for our government. The government picked the voters...
    - vwcx 6y ago
      As a DC resident, it does feel really frustrating, at a time when the nation's politics feel especially important, to not even have a representative to speak with/pressure. Especially when the chief argument against DC statehood has been that it'll tip the Senate to the left. BS.
abeppu 6y ago
Maybe someone who knows a bunch about this stuff can enlighten me. I was kind of under the impression that a large solar flare would damage _all sorts_ of electronic equipment, by inducing currents wires all over the place. - If we had extra transformers on hand, how would you protect them from also being damaged? - If a solar flare is powerful enough to damage these transformers, would it also have damaged a large proportion of devices that use power? I.e. even if you could get the electric grid working, would there be working systems of any complexity to use the power?
- Invictus0 6y ago
  I'm also not super knowledgeable in this area but let me have a shot: * The transformers could be stored in a manner designed to keep them from being damaged by this event (off? physically disconnected? In large pieces?) * I think things are surge protected at local levels so appliances would be fine.
- btilly 6y ago
  It is true that the solar flare will induce current in any exposed wire. However the following factors increase the current. The length of the wire, the thicker the wire, how poorly insulated it is, and the stronger the magnetic surges caused by the flare. Long distance transmission lines are very long, fairly thick, and very exposed. The result is that the surges that they experience are several orders of magnitude stronger than, say, inside of your washing machine or a transformer unconnected to the wire. That surge doesn't hurt the wire, but it can blow the transformer when it hits it. And hence the problem is mostly in blown transformers. If you have other transformers on hand, you swap them in and most of the system should be up.
  - makomk 6y ago
    I can see a couple of rather obvious problem with this. The electrical grid is, well, connected - the long, high-voltage sections are connected to the lower voltage sections which are connected to the even lower voltage sections. Those connections are through transformers, but if the surge is big enough to destroy even the largest transformers what's to stop it from blowing through all of the transformers and frying the delicate low-voltage control electronics of every grid connected device, including ones critical to operating the grid and power plants? Also, smaller devices are naturally going to be a lot more sensitive than huge transformers, so even if they don't experience the same level of voltage and current this could still be enough to destroy them.
PopeDotNinja 6y ago
Solar storm of 1859 https://en.wikipedia.org/wiki/Solar_storm_of_1859 https://en.wikipedia.org/wiki/Solar_storm_of_1859
rdxm 6y ago
curious as to how a post like this gets allowed. this is quite obviously click-bait, a personal blog post that is pretty dramatically un-informed. is HN going the route of Fox/CNN/DailyMail???
ars 6y ago
I wonder how many other rare disasters there are that cost only 300M to prepare for. If it's just a couple, then sure, fund this one. But if there are thousand of possible rare disasters - how can you possibly fund them all?
- dmurray 6y ago
  Now that we've got "global pandemic" marked off our bingo card, I'd put this at the top of the most likely "unforeseen" worldwide disasters. Some kind of supervolcano eruption is the other candidate. Of course, not all disasters are global and every country or region will have its own rare disasters to prepare for.
  - FlyMoreRockets 6y ago
    Add asteroid strike at with the energy level of a nuclear warhead to that list. Fortunately, it is 71% likely to come down in the ocean.
    - dmurray 6y ago
      It would have to be a lot bigger than "a nuclear warhead" to cause a global catastrophe rather than a local one, and it has to be more unfortunately aimed than just coming down over land. It would be pretty bad if the Chelyabinsk meteor had hit Tokyo or Mexico City, but as it happened it troubled glaziers more than doctors.
      - FlyMoreRockets 6y ago
        The problem with a sudden nuclear scale explosion is that it could trigger a nuclear exchange.
- williamdclt 6y ago
  If there's 3.3 thousand similar disasters, it would cost ~1 trillion to prepare. How much did covid-19 cost?
btilly 6y ago
How much should we worry about this? http://www.lloyds.com/~/media/lloyds/reports/emerging%20risk%20reports/solar%20storm%20risk%20to%20the%20north%20american%20electric%20grid.pdf http://www.lloyds.com/~/media/lloyds/reports/emerging%20risk... was prepared by Lloyds of London after the 2012 detection of a large solar flare by a satellite in interplanetary space. (That one missed the Earth.) Their estimate was $600 billion - $2 trillion dollars of damage from an event that happens roughly once ever 150 years on average. Therefore the amortized cost of this risk per year is $4 - $13 billion. If a substantial fraction of the risk can be mitigated over the next decade for $300 million, it would be cheap at 10x the price. In fact Berkshire Hathaway probably has enough exposure to this risk that it makes financial sense for them to not debate over who pays and to just create the stockpile to reduce potential future insurance claims.
- realtalk_sp 6y ago
  Sadly it might cost Berkshire even less to just reinsure against the risk.
  - phonon 6y ago
    Berkshire is the 5th largest reinsurer in the world[1], so not likely to happen... [1]https://www.insurancebusinessmag.com/us/news/breaking-news/these-are-the-top-10-reinsurance-companies-in-the-world-189205.aspx https://www.insurancebusinessmag.com/us/news/breaking-news/t...
    - realtalk_sp 6y ago
      Not sure what you're trying to imply. Reinsurers do in fact reinsure against their own risk. It's turtles all the way down.
      - phonon 6y ago
        Yes, retrocession agreements. I know. Berkshire Hathaway is not usually (ever?) on the ceding end of those arrangements.
dvt 6y ago
Being "prepared for black swans" is a bad idea. Of course, you want your system as a whole to be as antifragile as possible, but one should never prepare for black swans, but rather buy insurance (so to speak): hedge, diversify, minimize impact. For example, we can imagine a world where some other pandemic is being spread. One that, instead of attacking the lungs, attacks the kidneys. All of a sudden we'd need dialysis machines -- not ventilators. See how preparing for some specific black swan is misguided (as we can be hit by any other, equally unlikely, black swan event)? > In testimony before a Congressional Committee, it has been asserted that a prolonged collapse of this nation’s electrical grid—through starvation, disease, and societal collapse—could result in the death of up to 90% of the American population. Okay, this sounds like an interesting assertion. Can we back this up? Is this just alarmism? Where's the data? > Yes, $300 million dollars for a stockpile of 30 HV transformers is far too expensive to prevent 90% of the public dying and the rest reduced to cannibalism. I see. So now we're assuming some off-the-charts assertion, which is dubious at best[1], is absolutely true. The cannibalism is thrown in for extra flair. Gotta' get them clicks somehow. > Update: I forget to mention that I’m far more worried about accidental nuclear war, bioterrorism and AI run amok than I am about solar flares. Anyone that's seriously worried about "AI run amok" (whatever that means) doesn't understand the first thing about AI. [1] https://www.popularmechanics.com/military/weapons/a25883/north-korea-cant-kill-ninety-percent-of-americans/ https://www.popularmechanics.com/military/weapons/a25883/nor...
- jameslevy 6y ago
  It is worth worrying about "AI run amok" with the help of humans with a profit motive, or some other motive.
- elliekelly 6y ago
  Investing a nominal amount in strategic backup equipment that you might never need is hedging. An event that has a low probability of occurring but that (if it occurs) will be highly catastrophic is exactly the scenario that calls for hedging. Most critical systems have already diversified by building in redundancies (generators) and an insurance check wouldn’t be very useful for keeping the rest of us alive.
jerkstate 6y ago
Did this guy even talk to a power grid engineer before posting his opinion? Offline spares of long lead time equipment are always kept by utilities. The risk to the power grid is institutional knowledge, if something happens to the small group of greybeards who run each regional utility, they could bring in new qualified engineers but it would take time to understand the system (which would cause an outage that either wouldn't happen in the first place or would be short to become protracted). The other major risk is if they are kept from doing their jobs due to budget or political issues (see: PG&E)
- beckingz 6y ago
  The main issue is that ultra high voltage transformers are large, expensive, have long lead times (12-36 months!), and are built almost entirely in China. You're correct though, that the entire energy industry skews old and is at risk of losing a lot of knowledge as talent retires. The real risk is squirrels and trees though. Flora and fauna do not play well with transmission equipment.
  - antsar 6y ago
    > long lead times (12-36 months!) I was skeptical, but this seems right (at least circa 2012). Here's a source: https://www.energy.gov/sites/prod/files/Large%20Power%20Transformer%20Study%20-%20June%202012_0.pdf https://www.energy.gov/sites/prod/files/Large%20Power%20Tran... (summarized in the Concluding Remarks section)
  - mepiethree 6y ago
    yeah, squirrels are going to kill us all someday. here's a fun map of all the power failures caused by squirrels, as documentation of WW3 https://cybersquirrel1.com/ https://cybersquirrel1.com/
- mepiethree 6y ago
  > Did this guy even talk to a power grid engineer before posting his opinion? Seems like no? haha. To his credit, he admits he is not an expert and asks us to tell him why he is wrong. Unfortunately, when I read the comments, it seems like none of his blog readers are experts on the subject, either. So, maybe not "Monday Morning Quarterbacking," but certainly Armchair Quarterbacking.
wfbarks 6y ago
I feel like 2008 was an era of irresponsibility. Home buyers took out risky loans, banks sold these loans without accepting responsibility for how risky they where, investors purchased these loans without taking the responsibility to even see what kind of Loans where even in these mortgage backed securities. Ratings agencies abdicated this responsibility too. Goldman lied to AIG, and AIG didn't do the research to understand the credit default swaps they where selling. Central banks didn't accept responsibility for anything, saying "it was impossible to see the crisis coming". It doesn't appear that we have left this paradigm at all. The only group that has improved is the consumers, households have significantly cut back leverage since the 2008 crisis. But companies have not. Companies don't feel they have any responsibilities to prepare for these long tail situations, and governments don't seem capable of doing it. leaving it to central banks to throw trillions at the problem after the fact.
- solarkraft 6y ago
  I feel like most of the economy (through wrong regulation) is one of irresponsibility. In many places there is a high incentive to mess things up, but little incentive to fix them.
- oldsklgdfth 6y ago
  I would argue that the 10 years after 08' have also lacked responsibility. I think the secondary effects of the covid shutdown are going to highlight some very poor decisions. Mainly rallying the economy, rather than making it more robust. I'm interested in seeing what happens with housing specifically.
  - wfbarks 6y ago
    thats basically my point, Companies still aren't taking responsibility for anything and are running the companies to keep the share price up so the Exec team can collect on their options in 4 years and move on. Similarly the government seems completely incompetent, and willfully blind to the risks being taken.
    - 6y ago
smrk007 6y ago
Is anyone working on solving this problem?
Havoc 6y ago
I don't think it's true that the US is unprepared. Most major grids have documented procedure to do a blackstart. Which would be painful as hell but hardly 90% of population dies level painful. Author seems to be assuming more widespread damage I guess. In which case a handful of HV transformers aren't like to be the biggest headache. I'd imagine there is a hell of a lot more sensitive equipment out there than HV transformers. Internet, stock market, comms etc. https://en.wikipedia.org/wiki/Black_start https://en.wikipedia.org/wiki/Black_start
- rvnx 6y ago
  This article is underestimating also how creative can people can become under stressful situations. For example, converting car engines as power generators, fixing these High voltage transformers, etc. People adapt, they will build power generators based on whatever they find, industries will shift to make photovoltaic panels and electric production will become local, etc. Yes it's 90% of people dying if there is no electricity and everybody stays home waiting to die. 1 billion people in the world live without electricity. The solar flares are a minor risk compared to the main actual risk (war or sabotage), if there is a war, no matter how many High voltage transformers you have in stock, they are going to be destroyed.
runawaybottle 6y ago
I think it’s in our interest to have some kind of political Readiness Act that assesses events like this (Pandemic being in the list, along with this). Spending a few billion on it is pennies compared 2 trillion that we thus far have spent, not including the widespread economic cost. Might be worth jamming into the Green New deal.
8bitsrule 6y ago
It's a very complex topic. But clearly any country heavily invested in and reliant on technology must include a significant fraction of tech/science literate leaders ... an actual requirement for (half of?) any state's Congress-critters. To whatever extent 'terrorism' is a real threat (and not just an excuse to maintain the military budget), many other dangers like pandemics and an EMP event and climate change were outlined decades ago. F-35s and aircraft carriers and hundreds of military bases won't cut it against them. Nor will faith.
anovikov 6y ago
How likely is simultaneous failure of 9 out of 30 of them? Totally unlikely if you ask me. And some redundancy is built into the system, so it's not like failure of 3 will cause loss of power in 1/3 of the U.S. - most likely none at all (or maybe only scheduled blackouts in peak times in some of the places). In Cyprus, we had such an event in 2011, when 2000 tons of explosives Russian tried to smuggle into Syria, were intercepted by a U.S. Navy and redirected to be offloaded here (because the ship carrying them was formally registered here and flew a Cyprus flag - really being owned by the KGB), and these explosives went off in the summer heat, wrecking a power station that produced 70% of the electricity here. We only had about 2 hours of blackout per day and a recommendation to limit a/c use, for about a month. Power transmission is super resilient and relay protection is a whole science of it's own. Especially in the U.S. when there are many virtually independent, privately owned and loosely interconnected transmission systems, which puts it at a relative advantage.
- jessriedel 6y ago
  > how likely is simultaneous failure of 9 out of 30 of them? Totally unlikely if you ask me. It's quite possible because their failure probabilities aren't independent. A geomagnetic storm could take most of them out at once. Like respiratory pandemics, this is an event that happens every ~century. https://en.wikipedia.org/wiki/Geomagnetic_storm https://en.wikipedia.org/wiki/Geomagnetic_storm
  - anovikov 6y ago
    But it will be known well in advance and can be mitigated by simply shutting the grid down for a few hours. Only places which will experience difficulties are those having a large fraction of nuclear power since a reactor can't be quickly restarted once shut down, so power will restore only in some days.
    - jessriedel 6y ago
      Do you have a cite? My impression was that we do not have sufficient preparations in place to do this. (E.g., we may have satellites that can be taken as weak indicators, but they are not designed for this, and there is no clear decision authority starting from possibly ambiguous satellite data to prophylactically shutting down power for 300M people against their will. Hopefully, recent events have made clear that "leaders will get their act together and act decisively during an emergency" cannot be the default assumption.)
mepiethree 6y ago
If it were as easy as spending $300M, every utility company in the country would be falling over themselves to do it. Utility companies make their money by rate basing their capital expenditures. The rate you pay for electricity is set by the state, to guarantee a certain ROI for the utility on approved infrastructure. Utilities literally spend money to make money. I couldn't agree more that our electric grid needs more investment. I won't pretend to know better than the PUCs and the utilities about how that should be spent or what should be passed on to the ratepayer. The only thing I do know is that my company (Kevala) has mapped tens of thousands of substations, transmission lines, and HV transformers, so it seems we would have a lot more to back up than those 30. Good time for non-wires alternatives?
watertom 6y ago
The movement to run the government as a business is at the heart of a lot of these problems. Governments have a different risk model than a for profit business. For profit businessEs can’t in any way financially justify to it’s stock holders the spending of money to protect against something that might never happen. I ran DR & BC for a very large global firm, I reported to the Board, and I can tell you these people thought I was insane, even to protect against things like fires. “When was the last time you heard of a large office building having a major fire with more than a couple of casualties?” This was actually said to me when I was proposing to improve the company’s fire protection system. I got out voted and they “ saved” the money. They still haven’t had a fire, so as far as they are concerned I was stupid. Electric companies are necessary so they know if something does go horribly wrong they get bailed out.
yagurastation 6y ago
I'll just drop this book recommendation here, which I found to be an interesting read during this pandemic: Blackout: Tomorrow Will Be Too Late (2012) Marc Elsberg

303 points by rictic 6y ago

While I don't disagree that transformers... let alone key transformer.. are critical. By disabling 9 transformers you cannot take down “THE GRID”. You could certainly cause havoc and perhaps take down sections of a grid, perhaps a large one such as new york city. But these would be localized. While ensuring that critical components are local and available is important much of this is being done in the energy sector. It’s not to say there simply are no replacements. However, the energy sector, including transmission, is made up of over 6000 separate organizations. Most of which are not federally controlled or controlled by the local government. Put simply they are for-profit companies. They are critical infrastructure and regulated by the gov, but not a part of. Put short, this is a serious matter, but not in the way it is presented here. It would be a herculean effort to take out 1 let alone the suggested 9. Especially in a coordinated fashion. EVEN IF it was achieved.. This would only cause localized issues. Not takedown. “THE GRID” This is a much larger discussion but a few things to consider: 1. There is no singular "grid". It just does not work that way. There are thousands of generation and transmission companies throughout the US. They all maintain there own “grids”. There is connectivity to provide ways to connect grids to sell/buy/shed power. But these connections are controlled and can be simply… disconnected. 2. Without going on a major diatribe on the many different attack vectors and inherent vulnerabilities in the energy sector…. Suffice to say.. The easiest and most sure way to take out a transformer is physical. They would need to be destroyed or disabled. Digital/remote interference would require an immense effort and campaign… we are not talking breaching the 9 locations… we’re talking hacking millions upon millions of connected devices and coordinating load to take out a transformer or the grid as whole…. The coordination and effort necessary for either are staggering. 3. There are in fact spares and inventory… just not in the hands of the feds.

tonyedgecombe 6y ago
You seem to be assuming some sort of terrorist event. I would have thought the most likely reason for the whole grid to go down is another Carrington event.
- csecdaemon 6y ago
  I'm not assuming anything/anyone specifically but lean towards nation-state, anarchist, terror, etc. A natural disaster such as a flare, earthquake, fire, etc. are a different story. The radiation and electrical magnetic pulse from a solar flare could hamper equipment. However recent studies in the last few years by EPRI, INL, and ORNL actually show that much equipment would survive and equipment that did not would be localized.
  - FlyMoreRockets 6y ago
    Care to link to any of those studies? I'd love to read them.
    - csecdaemon 6y ago
      Here is EPRI report - https://www.epri.com/#/pages/product/3002014979/?lang=en-US https://www.epri.com/#/pages/product/3002014979/?lang=en-US
  - runawaybottle 6y ago
    Do we stress test our grid(s) core components with inputs that would be similar to that of a solar flare to write any of this off? I feel like 5 months ago we’d be sitting here dismissing the notion that a face mask shortage is a trivial problem, all countries can just suddenly ramp up production. But we couldn’t.
FpUser 6y ago
There is no singular "grid". It just does not work that way This did not prevent "Northeast blackout of 2003" when bunch of states in the US and province of Ontario in Canada went without power for 3 days. I remember it quite clearly. Was not big fun. Luckily my friends and I with families went camping on the lake so did not really suffer other than some little disaster in a fridge ;)
- runawaybottle 6y ago
  From wiki The blackout's proximate cause was a software bug in the alarm system at the control room of FirstEnergy, an Akron, Ohio–based company, which rendered operators unaware of the need to redistribute load after overloaded transmission lines drooped into foliage. What should have been a manageable local blackout cascaded into the collapse of the entire Northeast region. How fragile we are.
- makomk 6y ago
  That was a cascading failure where the instability of part of the grid tripped protection systems intended to prevent damage to equipment throughout the rest. The individual regions making up that grid were perfectly capable of operating independently, possibly with reduced capacity, even after the blackout. What they couldn't do is remain operational whilst unexpectedly being cut off, because that caused transient unbalances between generation and load. I'm pretty sure there's no failure of a small number of transformers that would leave the US as a whole, or any large chunk of it, without power for a prolonged period. There are probably scenarios where it could trigger a cascading failure where protection devices shut down the rest of the grid in order to protect the other equipment from imbalances caused by the failures, but that alone is not going to lead to 12-18 months of blackouts and the death of 90% of the US population.

xkapastel 6y ago

> In testimony before a Congressional Committee, it has been asserted that a prolonged collapse of this nation’s electrical grid—through starvation, disease, and societal collapse—could result in the death of up to 90% of the American population. This could happen at any time due to a solar flare. The coronavirus has made it pretty clear that America is completely unprepared for any sort of major deviation from "normal".

deleted 6y ago
[deleted]
intopieces 6y ago
> The coronavirus has made it pretty clear that America is completely unprepared for any sort of major deviation from "normal". This is the nature of the US system of competitive federalism. States only coordinate with the federal government when either (a) it's politically expedient, because the ruling party in the same for both at that time, or (b) the federal government forces this issue, at which time lawsuits ensue. It's helpful to think of the US as more a series of regions that sometimes fall under a shifting umbrella government than an actual, single entity.
- magicsmoke 6y ago
  It's pretty interesting that most countries in Europe are unitary states while most countries in the Americas are federal states. Maybe due to the centuries of warfare in Europe the states that survived were unitary states that could better handle external threats. Or maybe it's a holdover from a tradition of monarchy that the Americas never had. Interestingly, Germany is also a federal state. Yet we never seem to hear about federal regions of Germany disagreeing with each other and fighting with the Federal government like we hear about the US. Any Germans want to spill the tea on German inter-regional political drama?
  - runawaybottle 6y ago
    It can be argued that this it is part of the American political character because our origin story was shaped by the combination of disparate states, as in, we could have easily had colonies that sided with the British. It got cemented due to Slavery as we basically carved up the country between slaves states and free states, and ultimately fought a civil war to deal with this bifurcation. It’s been our long history, it defined us. Edit: Whoops, I did the American thing again and forgot there are other countries in the Americas.

PacketPaul 6y ago

The book “One Second After” deals with an EMP attack on the US. I don’t understand why we are not better prepared.

barkingcat 6y ago
because it costs money and decreases profits.
madengr 6y ago
We are, thats what the military and intelligence agencies do, and why we have a retaliatory nuclear arsenal.

wcoenen 6y ago

> I’d like to ask you guys whether we are prepared for other black swans. Let’s start with a collapse of the electrical system due to solar flares or electromagnetic pulse attacks. A Black Swan is a "high-profile, hard-to-predict, and rare event that is beyond the realm of normal expectations". But we know that the Carrington Event happened in 1859, and that it's only a matter of time before a big solar flare will affect Earth in the same way. So this is not a Black Swan. But yes, we should prepare for foreseeable rare events.

kiba 6y ago
A Black Swan is probably more about the POV of an actor as opposed to somebody who knows this, then it's not a black swan.
beambot 6y ago
Events like the Carrington & Covid seem to fit: high profile, hard to predict and rare. IMO, both qualify as Black swan. At some point, you have to pick a threshold on the probabilities involved... These events are likely independent & identically distributed (i.e. hard to predict for any year), and I'd call sub-1% annual chance pretty rare. But I'm receptive to a different threshold... But you can't just say "Black swan only counts for previously-inconceivable events". Edit: I'm assuming covid-19 is structurally different than Sars, mers, Ebola, & H1N1 purely based on the global effect it has had to date. If you found those others as pandemics (not according to WHO though), then you can remove it as Black swan.
- AQuantized 6y ago
  I think they are both categorically grey swans. https://www.investopedia.com/terms/g/grey-swan.asp https://www.investopedia.com/terms/g/grey-swan.asp COVID-19 especially because of the previous impact of Spanish Flu.
- Izkata 6y ago
  > These events are likely independent & identically distributed (i.e. hard to predict for any year), and I'd call sub-1% annual chance pretty rare. Put another way, it seems reasonable to me that "no one alive has experienced it before (and probably not their parents either)" is a decent threshold.

renewiltord 6y ago

Safrr Arkstorm or Tsunami are also pretty cheap to take care of. But that's life.

flipgimble 6y ago

This article has an unexpected resonance for me after several weeks of pandemic lockdown. Just three months ago I would have laughed it off. Now I am increasingly pissed off. In the US, but I see reflection of this in Europe as well, has allowed irresponsible puerile imbeciles to be elected. These people play games with wealth and power, with no regard to the lives and well-being of the society that pays for it. I did not mention any party, that perennial misdirection to their collective heist. Also don't fall for the other smoke screen of small vs. big government. I demand at least a competent government. For politicians truth is whatever you make people believe it to be. Meanwhile engineers and scientists struggle daily to find the hard facts of the world to painstakingly build or discover something that improves lives. We should be disgusted and revolted at who is leading us right now. We should drive these fools out office and strip them of any respect. If we don't, there will be no one competent left in position to rebuild from this, or the next, catastrophe that "nobody" could have expected. Or maybe there will be some states or localities that can get their act together, and the rest will be a wasteland of warring cultural and racial tribes.

twomoretime 6y ago
There's an old saying that people deserve their rulers.
- Afforess 6y ago
  This attitude is toxic and worse, incorrect. The majority of Americans did not vote for our President. I live in Austin, and my state district in TX includes a tiny slice of a city hundreds of miles away, Houston. Many state districts are like this, gerrymandered to prevent my vote from counting. Some places have it worse, DC, the capital, has no representation at all (fun trivia: Google the DC license plate). The voters didn't ask nor vote for our government. The government picked the voters...
  - vwcx 6y ago
    As a DC resident, it does feel really frustrating, at a time when the nation's politics feel especially important, to not even have a representative to speak with/pressure. Especially when the chief argument against DC statehood has been that it'll tip the Senate to the left. BS.

abeppu 6y ago

Maybe someone who knows a bunch about this stuff can enlighten me. I was kind of under the impression that a large solar flare would damage _all sorts_ of electronic equipment, by inducing currents wires all over the place. - If we had extra transformers on hand, how would you protect them from also being damaged? - If a solar flare is powerful enough to damage these transformers, would it also have damaged a large proportion of devices that use power? I.e. even if you could get the electric grid working, would there be working systems of any complexity to use the power?

Invictus0 6y ago
I'm also not super knowledgeable in this area but let me have a shot: * The transformers could be stored in a manner designed to keep them from being damaged by this event (off? physically disconnected? In large pieces?) * I think things are surge protected at local levels so appliances would be fine.
btilly 6y ago
It is true that the solar flare will induce current in any exposed wire. However the following factors increase the current. The length of the wire, the thicker the wire, how poorly insulated it is, and the stronger the magnetic surges caused by the flare. Long distance transmission lines are very long, fairly thick, and very exposed. The result is that the surges that they experience are several orders of magnitude stronger than, say, inside of your washing machine or a transformer unconnected to the wire. That surge doesn't hurt the wire, but it can blow the transformer when it hits it. And hence the problem is mostly in blown transformers. If you have other transformers on hand, you swap them in and most of the system should be up.
- makomk 6y ago
  I can see a couple of rather obvious problem with this. The electrical grid is, well, connected - the long, high-voltage sections are connected to the lower voltage sections which are connected to the even lower voltage sections. Those connections are through transformers, but if the surge is big enough to destroy even the largest transformers what's to stop it from blowing through all of the transformers and frying the delicate low-voltage control electronics of every grid connected device, including ones critical to operating the grid and power plants? Also, smaller devices are naturally going to be a lot more sensitive than huge transformers, so even if they don't experience the same level of voltage and current this could still be enough to destroy them.

PopeDotNinja 6y ago

Solar storm of 1859 https://en.wikipedia.org/wiki/Solar_storm_of_1859 https://en.wikipedia.org/wiki/Solar_storm_of_1859

rdxm 6y ago

curious as to how a post like this gets allowed. this is quite obviously click-bait, a personal blog post that is pretty dramatically un-informed. is HN going the route of Fox/CNN/DailyMail???

ars 6y ago

I wonder how many other rare disasters there are that cost only 300M to prepare for. If it's just a couple, then sure, fund this one. But if there are thousand of possible rare disasters - how can you possibly fund them all?

dmurray 6y ago
Now that we've got "global pandemic" marked off our bingo card, I'd put this at the top of the most likely "unforeseen" worldwide disasters. Some kind of supervolcano eruption is the other candidate. Of course, not all disasters are global and every country or region will have its own rare disasters to prepare for.
- FlyMoreRockets 6y ago
  Add asteroid strike at with the energy level of a nuclear warhead to that list. Fortunately, it is 71% likely to come down in the ocean.
  - dmurray 6y ago
    It would have to be a lot bigger than "a nuclear warhead" to cause a global catastrophe rather than a local one, and it has to be more unfortunately aimed than just coming down over land. It would be pretty bad if the Chelyabinsk meteor had hit Tokyo or Mexico City, but as it happened it troubled glaziers more than doctors.
    - FlyMoreRockets 6y ago
      The problem with a sudden nuclear scale explosion is that it could trigger a nuclear exchange.
williamdclt 6y ago
If there's 3.3 thousand similar disasters, it would cost ~1 trillion to prepare. How much did covid-19 cost?

btilly 6y ago

How much should we worry about this? http://www.lloyds.com/~/media/lloyds/reports/emerging%20risk%20reports/solar%20storm%20risk%20to%20the%20north%20american%20electric%20grid.pdf http://www.lloyds.com/~/media/lloyds/reports/emerging%20risk... was prepared by Lloyds of London after the 2012 detection of a large solar flare by a satellite in interplanetary space. (That one missed the Earth.) Their estimate was $600 billion - $2 trillion dollars of damage from an event that happens roughly once ever 150 years on average. Therefore the amortized cost of this risk per year is $4 - $13 billion. If a substantial fraction of the risk can be mitigated over the next decade for $300 million, it would be cheap at 10x the price. In fact Berkshire Hathaway probably has enough exposure to this risk that it makes financial sense for them to not debate over who pays and to just create the stockpile to reduce potential future insurance claims.

realtalk_sp 6y ago
Sadly it might cost Berkshire even less to just reinsure against the risk.
- phonon 6y ago
  Berkshire is the 5th largest reinsurer in the world[1], so not likely to happen... [1]https://www.insurancebusinessmag.com/us/news/breaking-news/these-are-the-top-10-reinsurance-companies-in-the-world-189205.aspx https://www.insurancebusinessmag.com/us/news/breaking-news/t...
  - realtalk_sp 6y ago
    Not sure what you're trying to imply. Reinsurers do in fact reinsure against their own risk. It's turtles all the way down.
    - phonon 6y ago
      Yes, retrocession agreements. I know. Berkshire Hathaway is not usually (ever?) on the ceding end of those arrangements.

dvt 6y ago

Being "prepared for black swans" is a bad idea. Of course, you want your system as a whole to be as antifragile as possible, but one should never prepare for black swans, but rather buy insurance (so to speak): hedge, diversify, minimize impact. For example, we can imagine a world where some other pandemic is being spread. One that, instead of attacking the lungs, attacks the kidneys. All of a sudden we'd need dialysis machines -- not ventilators. See how preparing for some specific black swan is misguided (as we can be hit by any other, equally unlikely, black swan event)? > In testimony before a Congressional Committee, it has been asserted that a prolonged collapse of this nation’s electrical grid—through starvation, disease, and societal collapse—could result in the death of up to 90% of the American population. Okay, this sounds like an interesting assertion. Can we back this up? Is this just alarmism? Where's the data? > Yes, $300 million dollars for a stockpile of 30 HV transformers is far too expensive to prevent 90% of the public dying and the rest reduced to cannibalism. I see. So now we're assuming some off-the-charts assertion, which is dubious at best[1], is absolutely true. The cannibalism is thrown in for extra flair. Gotta' get them clicks somehow. > Update: I forget to mention that I’m far more worried about accidental nuclear war, bioterrorism and AI run amok than I am about solar flares. Anyone that's seriously worried about "AI run amok" (whatever that means) doesn't understand the first thing about AI. [1] https://www.popularmechanics.com/military/weapons/a25883/north-korea-cant-kill-ninety-percent-of-americans/ https://www.popularmechanics.com/military/weapons/a25883/nor...

jameslevy 6y ago
It is worth worrying about "AI run amok" with the help of humans with a profit motive, or some other motive.
elliekelly 6y ago
Investing a nominal amount in strategic backup equipment that you might never need is hedging. An event that has a low probability of occurring but that (if it occurs) will be highly catastrophic is exactly the scenario that calls for hedging. Most critical systems have already diversified by building in redundancies (generators) and an insurance check wouldn’t be very useful for keeping the rest of us alive.

jerkstate 6y ago

Did this guy even talk to a power grid engineer before posting his opinion? Offline spares of long lead time equipment are always kept by utilities. The risk to the power grid is institutional knowledge, if something happens to the small group of greybeards who run each regional utility, they could bring in new qualified engineers but it would take time to understand the system (which would cause an outage that either wouldn't happen in the first place or would be short to become protracted). The other major risk is if they are kept from doing their jobs due to budget or political issues (see: PG&E)

beckingz 6y ago
The main issue is that ultra high voltage transformers are large, expensive, have long lead times (12-36 months!), and are built almost entirely in China. You're correct though, that the entire energy industry skews old and is at risk of losing a lot of knowledge as talent retires. The real risk is squirrels and trees though. Flora and fauna do not play well with transmission equipment.
- antsar 6y ago
  > long lead times (12-36 months!) I was skeptical, but this seems right (at least circa 2012). Here's a source: https://www.energy.gov/sites/prod/files/Large%20Power%20Transformer%20Study%20-%20June%202012_0.pdf https://www.energy.gov/sites/prod/files/Large%20Power%20Tran... (summarized in the Concluding Remarks section)
- mepiethree 6y ago
  yeah, squirrels are going to kill us all someday. here's a fun map of all the power failures caused by squirrels, as documentation of WW3 https://cybersquirrel1.com/ https://cybersquirrel1.com/
mepiethree 6y ago
> Did this guy even talk to a power grid engineer before posting his opinion? Seems like no? haha. To his credit, he admits he is not an expert and asks us to tell him why he is wrong. Unfortunately, when I read the comments, it seems like none of his blog readers are experts on the subject, either. So, maybe not "Monday Morning Quarterbacking," but certainly Armchair Quarterbacking.

wfbarks 6y ago

I feel like 2008 was an era of irresponsibility. Home buyers took out risky loans, banks sold these loans without accepting responsibility for how risky they where, investors purchased these loans without taking the responsibility to even see what kind of Loans where even in these mortgage backed securities. Ratings agencies abdicated this responsibility too. Goldman lied to AIG, and AIG didn't do the research to understand the credit default swaps they where selling. Central banks didn't accept responsibility for anything, saying "it was impossible to see the crisis coming". It doesn't appear that we have left this paradigm at all. The only group that has improved is the consumers, households have significantly cut back leverage since the 2008 crisis. But companies have not. Companies don't feel they have any responsibilities to prepare for these long tail situations, and governments don't seem capable of doing it. leaving it to central banks to throw trillions at the problem after the fact.

solarkraft 6y ago
I feel like most of the economy (through wrong regulation) is one of irresponsibility. In many places there is a high incentive to mess things up, but little incentive to fix them.
oldsklgdfth 6y ago
I would argue that the 10 years after 08' have also lacked responsibility. I think the secondary effects of the covid shutdown are going to highlight some very poor decisions. Mainly rallying the economy, rather than making it more robust. I'm interested in seeing what happens with housing specifically.
- wfbarks 6y ago
  thats basically my point, Companies still aren't taking responsibility for anything and are running the companies to keep the share price up so the Exec team can collect on their options in 4 years and move on. Similarly the government seems completely incompetent, and willfully blind to the risks being taken.
  - 6y ago

smrk007 6y ago

Is anyone working on solving this problem?

Havoc 6y ago

I don't think it's true that the US is unprepared. Most major grids have documented procedure to do a blackstart. Which would be painful as hell but hardly 90% of population dies level painful. Author seems to be assuming more widespread damage I guess. In which case a handful of HV transformers aren't like to be the biggest headache. I'd imagine there is a hell of a lot more sensitive equipment out there than HV transformers. Internet, stock market, comms etc. https://en.wikipedia.org/wiki/Black_start https://en.wikipedia.org/wiki/Black_start

rvnx 6y ago
This article is underestimating also how creative can people can become under stressful situations. For example, converting car engines as power generators, fixing these High voltage transformers, etc. People adapt, they will build power generators based on whatever they find, industries will shift to make photovoltaic panels and electric production will become local, etc. Yes it's 90% of people dying if there is no electricity and everybody stays home waiting to die. 1 billion people in the world live without electricity. The solar flares are a minor risk compared to the main actual risk (war or sabotage), if there is a war, no matter how many High voltage transformers you have in stock, they are going to be destroyed.

runawaybottle 6y ago

I think it’s in our interest to have some kind of political Readiness Act that assesses events like this (Pandemic being in the list, along with this). Spending a few billion on it is pennies compared 2 trillion that we thus far have spent, not including the widespread economic cost. Might be worth jamming into the Green New deal.

8bitsrule 6y ago

It's a very complex topic. But clearly any country heavily invested in and reliant on technology must include a significant fraction of tech/science literate leaders ... an actual requirement for (half of?) any state's Congress-critters. To whatever extent 'terrorism' is a real threat (and not just an excuse to maintain the military budget), many other dangers like pandemics and an EMP event and climate change were outlined decades ago. F-35s and aircraft carriers and hundreds of military bases won't cut it against them. Nor will faith.

anovikov 6y ago

How likely is simultaneous failure of 9 out of 30 of them? Totally unlikely if you ask me. And some redundancy is built into the system, so it's not like failure of 3 will cause loss of power in 1/3 of the U.S. - most likely none at all (or maybe only scheduled blackouts in peak times in some of the places). In Cyprus, we had such an event in 2011, when 2000 tons of explosives Russian tried to smuggle into Syria, were intercepted by a U.S. Navy and redirected to be offloaded here (because the ship carrying them was formally registered here and flew a Cyprus flag - really being owned by the KGB), and these explosives went off in the summer heat, wrecking a power station that produced 70% of the electricity here. We only had about 2 hours of blackout per day and a recommendation to limit a/c use, for about a month. Power transmission is super resilient and relay protection is a whole science of it's own. Especially in the U.S. when there are many virtually independent, privately owned and loosely interconnected transmission systems, which puts it at a relative advantage.

jessriedel 6y ago
> how likely is simultaneous failure of 9 out of 30 of them? Totally unlikely if you ask me. It's quite possible because their failure probabilities aren't independent. A geomagnetic storm could take most of them out at once. Like respiratory pandemics, this is an event that happens every ~century. https://en.wikipedia.org/wiki/Geomagnetic_storm https://en.wikipedia.org/wiki/Geomagnetic_storm
- anovikov 6y ago
  But it will be known well in advance and can be mitigated by simply shutting the grid down for a few hours. Only places which will experience difficulties are those having a large fraction of nuclear power since a reactor can't be quickly restarted once shut down, so power will restore only in some days.
  - jessriedel 6y ago
    Do you have a cite? My impression was that we do not have sufficient preparations in place to do this. (E.g., we may have satellites that can be taken as weak indicators, but they are not designed for this, and there is no clear decision authority starting from possibly ambiguous satellite data to prophylactically shutting down power for 300M people against their will. Hopefully, recent events have made clear that "leaders will get their act together and act decisively during an emergency" cannot be the default assumption.)

mepiethree 6y ago

If it were as easy as spending $300M, every utility company in the country would be falling over themselves to do it. Utility companies make their money by rate basing their capital expenditures. The rate you pay for electricity is set by the state, to guarantee a certain ROI for the utility on approved infrastructure. Utilities literally spend money to make money. I couldn't agree more that our electric grid needs more investment. I won't pretend to know better than the PUCs and the utilities about how that should be spent or what should be passed on to the ratepayer. The only thing I do know is that my company (Kevala) has mapped tens of thousands of substations, transmission lines, and HV transformers, so it seems we would have a lot more to back up than those 30. Good time for non-wires alternatives?

watertom 6y ago

The movement to run the government as a business is at the heart of a lot of these problems. Governments have a different risk model than a for profit business. For profit businessEs can’t in any way financially justify to it’s stock holders the spending of money to protect against something that might never happen. I ran DR & BC for a very large global firm, I reported to the Board, and I can tell you these people thought I was insane, even to protect against things like fires. “When was the last time you heard of a large office building having a major fire with more than a couple of casualties?” This was actually said to me when I was proposing to improve the company’s fire protection system. I got out voted and they “ saved” the money. They still haven’t had a fire, so as far as they are concerned I was stupid. Electric companies are necessary so they know if something does go horribly wrong they get bailed out.

yagurastation 6y ago

I'll just drop this book recommendation here, which I found to be an interesting read during this pandemic: Blackout: Tomorrow Will Be Too Late (2012) Marc Elsberg