6 ms·

I don't know the first thing about electric infrastructure, but for there to be a single point of failure like this just seems bonkers. I suppose the country's

I don't know the first thing about electric infrastructure, but for there to be a single point of failure like this just seems bonkers. I suppose the country's shape is partially to blame. I wonder what could have caused this.

cmrdporcupine 2y ago
Are you old enough to remember the great northeast blackout of 2003? A single software bug brought the whole grid down for 55 million people and we didn't have power fully restored here in southern Ontario until 3 days later. It was nice to see the stars though.
- greenie_beans 2y ago
  just gave me a good ambiance idea for a fiction, ty
  - PotatoNinja 2y ago
    https://www.amazon.com/Blackout-heart-stopping-techno-thriller-Marc-Elsberg-ebook/dp/B01MYDPTLR https://www.amazon.com/Blackout-heart-stopping-techno-thrill...
    - greenie_beans 2y ago
      nope nothing like that at all, like so far away from what i'm thinking. i'm writing what i'm calling "the walking book" about a guy and his dog walking everyday over a lifetime, where the tension of the plot is about all the things that happen throughout all of the different walks
- jcranmer 2y ago
  It wasn't a single software bug. Broadly, what happened was this: * A power company failed to trim trees from its power lines properly. Four of their major transmission lines failed in one afternoon due to shorting out via tree. * The software bug you mentioned caused a failure to alarm the power company of two of their line failures. The first and fourth failures were alarmed in real time. * A separate issue rendered the regional grid operator's software modeling real-time grid instability inoperable for most of the afternoon. Crucially, if this had been running, the operator would have realized that the failure of one more line would have created grid instability requiring immediate action. * The final line trip set off a cascade of overloads trips until Sammis-Star overloads and trips. This takes about 15 minutes, and analysis suggests that the Sammis-Star trip is when the blackout became inevitable. * Sammis-Star triggers lines to fail one-by-one from east to west, until it severed every line in Ohio and Michigan. This causes a large power surge to go from Michigan through southern Ohio, Pennsylvania, New York, Ontario, back into Michigan and then into the final demand of Cleveland. * Essentially every link along this spiral fails in the course of a few seconds, creating several grid islands. Whether these grid islands black out is dependent on the local mismatch between generation and consumption. (Full story from https://web.archive.org/web/20120318081212/http://www.nerc.com/docs/docs/blackout/NERC_Final_Blackout_Report_07_13_04.pdf https://web.archive.org/web/20120318081212/http://www.nerc.c...).
Maxion 2y ago
Electricity infrastructure is incredibly expensive. If a main transmission line goes down you will have issues both sides of it, one side experiencing sudden increased supply which will raise the frequency which will cause power plants to trip offline. On the other side you will see frequency drops, which will do the same. I do not know what Chiles power production looks like, but it would be a huge challenge for any power network to deal with one of the main lines suddenly dropping out.
toomuchtodo 2y ago
Broadly speaking, electrical grids are the largest machines in the world and are held together by systems to keep everything in harmony at a target frequency (enormous spinning mass that slows when load is added, and has to spin back up, failsafes that have to decouple parts of the grid when maintaining voltage and frequency becomes impossible with committed capacity, etc). It’s actually wild it works flawlessly most of the time and is a testament to robust systems keeping them operational. There’s always room for improvement between here and some nebulous point of diminishing returns. That cost/resiliency discovery process is constant. In this case, there is room for improvement, at some cost to be determined from a post mortem.
- Galatians4_16 2y ago
  Giant flywheels exist, in some systems, which can easily start or restart backup diesel generators on the fly, though probably not for whole grids. A bit of decentralization goes a long way.
  - toomuchtodo 2y ago
    Batteries are taking over this role. Tesla has one near Houston, TX that can have an isolated electrical path established from it to geographically close thermal generators to provide blackstart capabilities (like jumpstarting a car). “Gambit Energy” is the project. To your point, distribution and grid segmentation solves for this, but it takes time and money to deploy batteries throughout a service territory, configure orchestration and transmission grade disconnects to rapidly isolate system components, etc. I see a large order of battery storage from BYD in Chile’s future. https://www.gambit-energystorage.com/ https://www.gambit-energystorage.com/ https://www.angleton.tx.us/DocumentCenter/View/3793/Gambit-Energy-Storage-Park-FAQ?bidId= https://www.angleton.tx.us/DocumentCenter/View/3793/Gambit-E... https://www.ercot.com/files/docs/2024/01/12/Tesla%20BESS%20Grid%20Forming%20and%20Augmentation_ERCOT.pdf https://www.ercot.com/files/docs/2024/01/12/Tesla%20BESS%20G... https://www.nrel.gov/grid/black-start.html https://www.nrel.gov/grid/black-start.html
    - no_wizard 2y ago
oliwarner 2y ago
It's likely not a single fault, but a fault and a subsequently failed protection mechanism. Elements of a grid usually work together, tied through frequency synchronisation with protection breakers to separate them if a part gets overloaded or fails. If a required generator or transmissions like did go down, and it and its users remained connected to the national grid, that load can very easily pull the rest down. Many modern countries have experienced rolling black- and brown-outs in the past ~25 years. It's not an easy thing to sidestep without a lot of spare capacity and that takes money that nobody has the appetite to spend.
VBprogrammer 2y ago
In general failures like this aren't so much because of a single point of failure but because they trigger cascading failures across the network. One piece of infrastructure trips offline, causing an abnormal situation at another, which then trips etc.
- ziofill 2y ago
  Isn’t this the same as a single point of failure, just “spread out”?
  - ffsm8 2y ago
    No, a single point of failure means that everything depends on a single thing. I.e. if you've got 10 gas engines all being fed by a single pipeline. Or a single powerline providing all electricity to the full grid. That pipeline/powerline is your single point of failure. It is still a single network, but you need multiple failures to manifest into a cascading issue like this. What you're probably thinking of is more akin to distributed system which fails when any one component fails, i.e. modern microservice architectures, aka distributed monoliths. But that's not the case here, because you constantly have minor issues on the grid. They're just continuously being handled. What becomes the issue is the cascade, with each failure increasing the likelihood of a following failure etc. In web developer terms, this is as if a production k8s cluster fails because a node went offline, which rebalanced too many containers to another node which ran out of memory, causing it to crash and starting the cascade, ultimately ending with adjacent clusters starting to crash because of the error quotas etc
    - ziofill 2y ago
      You are right, I was thinking of a distributed system, it’s not the same. But it’s still risky and brittle.
  - tharkun__ 2y ago
    You'll want to read timelines of blackouts, e.g. this one: https://en.wikipedia.org/wiki/Northeast_blackout_of_2003#Timeline https://en.wikipedia.org/wiki/Northeast_blackout_of_2003#Tim... It starts with a small thing at 12:15 p.m. and takes a while for other things to happen. But then bam 4:05:57 p.m. to 4:13 p.m. results in: End of cascading failure. 256 power plants are off-line, 85% of which went offline after the grid separations occurred, most due to the action of automatic protective controls.

cmrdporcupine 2y ago

Are you old enough to remember the great northeast blackout of 2003? A single software bug brought the whole grid down for 55 million people and we didn't have power fully restored here in southern Ontario until 3 days later. It was nice to see the stars though.

greenie_beans 2y ago
just gave me a good ambiance idea for a fiction, ty
- PotatoNinja 2y ago
  https://www.amazon.com/Blackout-heart-stopping-techno-thriller-Marc-Elsberg-ebook/dp/B01MYDPTLR https://www.amazon.com/Blackout-heart-stopping-techno-thrill...
  - greenie_beans 2y ago
    nope nothing like that at all, like so far away from what i'm thinking. i'm writing what i'm calling "the walking book" about a guy and his dog walking everyday over a lifetime, where the tension of the plot is about all the things that happen throughout all of the different walks
jcranmer 2y ago
It wasn't a single software bug. Broadly, what happened was this: * A power company failed to trim trees from its power lines properly. Four of their major transmission lines failed in one afternoon due to shorting out via tree. * The software bug you mentioned caused a failure to alarm the power company of two of their line failures. The first and fourth failures were alarmed in real time. * A separate issue rendered the regional grid operator's software modeling real-time grid instability inoperable for most of the afternoon. Crucially, if this had been running, the operator would have realized that the failure of one more line would have created grid instability requiring immediate action. * The final line trip set off a cascade of overloads trips until Sammis-Star overloads and trips. This takes about 15 minutes, and analysis suggests that the Sammis-Star trip is when the blackout became inevitable. * Sammis-Star triggers lines to fail one-by-one from east to west, until it severed every line in Ohio and Michigan. This causes a large power surge to go from Michigan through southern Ohio, Pennsylvania, New York, Ontario, back into Michigan and then into the final demand of Cleveland. * Essentially every link along this spiral fails in the course of a few seconds, creating several grid islands. Whether these grid islands black out is dependent on the local mismatch between generation and consumption. (Full story from https://web.archive.org/web/20120318081212/http://www.nerc.com/docs/docs/blackout/NERC_Final_Blackout_Report_07_13_04.pdf https://web.archive.org/web/20120318081212/http://www.nerc.c...).

Maxion 2y ago

Electricity infrastructure is incredibly expensive. If a main transmission line goes down you will have issues both sides of it, one side experiencing sudden increased supply which will raise the frequency which will cause power plants to trip offline. On the other side you will see frequency drops, which will do the same. I do not know what Chiles power production looks like, but it would be a huge challenge for any power network to deal with one of the main lines suddenly dropping out.

toomuchtodo 2y ago

Broadly speaking, electrical grids are the largest machines in the world and are held together by systems to keep everything in harmony at a target frequency (enormous spinning mass that slows when load is added, and has to spin back up, failsafes that have to decouple parts of the grid when maintaining voltage and frequency becomes impossible with committed capacity, etc). It’s actually wild it works flawlessly most of the time and is a testament to robust systems keeping them operational. There’s always room for improvement between here and some nebulous point of diminishing returns. That cost/resiliency discovery process is constant. In this case, there is room for improvement, at some cost to be determined from a post mortem.

Galatians4_16 2y ago
Giant flywheels exist, in some systems, which can easily start or restart backup diesel generators on the fly, though probably not for whole grids. A bit of decentralization goes a long way.
- toomuchtodo 2y ago
  Batteries are taking over this role. Tesla has one near Houston, TX that can have an isolated electrical path established from it to geographically close thermal generators to provide blackstart capabilities (like jumpstarting a car). “Gambit Energy” is the project. To your point, distribution and grid segmentation solves for this, but it takes time and money to deploy batteries throughout a service territory, configure orchestration and transmission grade disconnects to rapidly isolate system components, etc. I see a large order of battery storage from BYD in Chile’s future. https://www.gambit-energystorage.com/ https://www.gambit-energystorage.com/ https://www.angleton.tx.us/DocumentCenter/View/3793/Gambit-Energy-Storage-Park-FAQ?bidId= https://www.angleton.tx.us/DocumentCenter/View/3793/Gambit-E... https://www.ercot.com/files/docs/2024/01/12/Tesla%20BESS%20Grid%20Forming%20and%20Augmentation_ERCOT.pdf https://www.ercot.com/files/docs/2024/01/12/Tesla%20BESS%20G... https://www.nrel.gov/grid/black-start.html https://www.nrel.gov/grid/black-start.html
  - no_wizard 2y ago

oliwarner 2y ago

It's likely not a single fault, but a fault and a subsequently failed protection mechanism. Elements of a grid usually work together, tied through frequency synchronisation with protection breakers to separate them if a part gets overloaded or fails. If a required generator or transmissions like did go down, and it and its users remained connected to the national grid, that load can very easily pull the rest down. Many modern countries have experienced rolling black- and brown-outs in the past ~25 years. It's not an easy thing to sidestep without a lot of spare capacity and that takes money that nobody has the appetite to spend.

VBprogrammer 2y ago

In general failures like this aren't so much because of a single point of failure but because they trigger cascading failures across the network. One piece of infrastructure trips offline, causing an abnormal situation at another, which then trips etc.

ziofill 2y ago
Isn’t this the same as a single point of failure, just “spread out”?
- ffsm8 2y ago
  No, a single point of failure means that everything depends on a single thing. I.e. if you've got 10 gas engines all being fed by a single pipeline. Or a single powerline providing all electricity to the full grid. That pipeline/powerline is your single point of failure. It is still a single network, but you need multiple failures to manifest into a cascading issue like this. What you're probably thinking of is more akin to distributed system which fails when any one component fails, i.e. modern microservice architectures, aka distributed monoliths. But that's not the case here, because you constantly have minor issues on the grid. They're just continuously being handled. What becomes the issue is the cascade, with each failure increasing the likelihood of a following failure etc. In web developer terms, this is as if a production k8s cluster fails because a node went offline, which rebalanced too many containers to another node which ran out of memory, causing it to crash and starting the cascade, ultimately ending with adjacent clusters starting to crash because of the error quotas etc
  - ziofill 2y ago
    You are right, I was thinking of a distributed system, it’s not the same. But it’s still risky and brittle.
- tharkun__ 2y ago
  You'll want to read timelines of blackouts, e.g. this one: https://en.wikipedia.org/wiki/Northeast_blackout_of_2003#Timeline https://en.wikipedia.org/wiki/Northeast_blackout_of_2003#Tim... It starts with a small thing at 12:15 p.m. and takes a while for other things to happen. But then bam 4:05:57 p.m. to 4:13 p.m. results in: End of cascading failure. 256 power plants are off-line, 85% of which went offline after the grid separations occurred, most due to the action of automatic protective controls.