5 ms·

The fact that there is not a single root cause but several ones makes me instinctively think this is a good report, because it's not what the "bosses" (and even

The fact that there is not a single root cause but several ones makes me instinctively think this is a good report, because it's not what the "bosses" (and even less politicians) like to hear.

ragebol 7mo ago
Yep, sounds like "This was bound to happen at some point"
- cucumber3732842 7mo ago
  Which on some level is exactly "what the bosses and politicians want to hear" When it's everybody's fault it's nobody's fault.
  - drob518 7mo ago
    Exactly.
  - darkwater 7mo ago
    In some ways, yes, but yet it's what reality is. There was probably some last factor kicking in that triggered the cascade, but there were probably many non-happy-paths not properly covered by working backup/fallback strategies. So a report could totally still tell "it's X fault", pointing the finger there. Government would blame the owner of X, some public statement about fixing X would be made and then the ones working in the field should internally push toi improve/fix their own (reduced) scope. I don't know what will come of this report in the next months/years, I will keep an eye on it though, since I live in Spain :)
  - lyu07282 7mo ago
    But EU's liberalized energy market gives us resiliency and low prices for electricity! /s
    - galbar 7mo ago
      But not across the Pyrenees :_)
drob518 7mo ago
Frequently, when you see these massive failures, the root cause is an alignment of small weaknesses that all come together on a specific day. See, for instance, the space shuttle O-ring incident, Three-Mile Island, Fukushima, etc. These are complex systems with lots of moving parts and lots of (sometimes independent) people managing them. In a sense, the complexity it the common root cause.
- amelius 7mo ago
  It usually starts with a broken coffee machine.
  - drob518 7mo ago
    When that happens, get ready.
- linuxguy2 7mo ago
  It's like the Swiss Cheese model where every system has "holes" or vulnerabilities, several layers, and a major incident only occurs when a hole aligns through all the layers. https://en.wikipedia.org/wiki/Swiss_cheese_model https://en.wikipedia.org/wiki/Swiss_cheese_model
  - Ringz 7mo ago
    I use this model all the time. It's very helpful for explaining the multifactorial genesis of catastrophes to ordinary people.
    - anonymars 7mo ago
      Also perhaps worth a read: https://devblogs.microsoft.com/oldnewthing/20080416-00/?p=22723#:~:text=You%E2%80%99ve%20all%20experienced,errors%20are%20exposed. https://devblogs.microsoft.com/oldnewthing/20080416-00/?p=22... "You’ve all experienced the Fundamental Failure-Mode Theorem: You’re investigating a problem and along the way you find some function that never worked. A cache has a bug that results in cache misses when there should be hits. A request for an object that should be there somehow always fails. And yet the system still worked in spite of these errors. Eventually you trace the problem to a recent change that exposed all of the other bugs. Those bugs were always there, but the system kept on working because there was enough redundancy that one component was able to compensate for the failure of another component. Sometimes this chain of errors and compensation continues for several cycles, until finally the last protective layer fails and the underlying errors are exposed."
OgsyedIE 7mo ago
There are ways to aggregate these into a single resilience score for policy makers with only moderate loss of detail but it's unpopular.
red_admiral 7mo ago
Yes, a lot of modern engineering is good enough that single-cause failures are very rare indeed. That means that failures themselves are rare, but when they do happen, they're most likely to have multiple causes. How to explain that to non-engineers is another problem.
- pseudohadamard 7mo ago
  I think a better way of explaining it to people is that we've made critical systems so reliable that, in order for them to fail, the failures have to be quite complex.
- javier2 7mo ago
  we are not as complicated as the national grid, I have been here for nearly 10 years now, and our outages have gone from single cause, two causes, or now its nearly always 3 things that need to go wrong at the same time.
- cameldrv 7mo ago
  This is almost universal in aviation. They always talk about the "accident chain." Essentially everything that can kill you with one mistake is illegal through training and operational requirements and engineering and maintenance regulations.
toomuchtodo 7mo ago
They need more battery storage for grid health, both colocated at solar PV generators (to buffer voltage and frequency anomalies) and spread throughout the grid. This replaces inertia and other grid services provided by spinning thermal generators. There was no market mechanism to encourage the deployment of this technology in concert with Spain’s rapid deployment of solar and wind.
- z2 7mo ago
  There are non-battery buffers available too--I recently got rooftop residential solar installed, and learned that my area is covered by a grid profile requiring that the solar system stay online through something like 60 +/- 2Hz before shutting down completely, and ramping down production linearly beyond a 1Hz deviation or so. The point is to avoid cascading shutdowns by riding through over/undersupply situations, whereas an older standard for my area would have the all solar systems cut off the moment frequency exceeded 60.5Hz (which would indicate oversupply from power plant generators spinning faster via lower resistance). In my system's case, switching to this grid profile was just a software toggle.
  - toomuchtodo 7mo ago
    This is grid following, very effective for small scale generation. It does not work for large scale generation though when the grid is relying on that voltage and frequency from the utility scale renewable generation ("grid forming"). When those large generators exceed their ride through tolerance, batteries step in to hold voltage and frequency up until the transient event ends or dispatchable generators called upon spin up (currently fossil gas primarily, but also nuclear if there is headroom to increase output). Thermal generators can take minutes to provide this support (called upon, fuel intake increased, spinning metal spins faster), batteries respond within 250-500ms. Tesla’s Megapack system at the Hornsdale Power Reserve in Australia was the first example of this being proven out at scale in prod. Batteries everywhere, as quickly as possible.
- tuetuopay 7mo ago
  In this very specific case, battery storage would not have helped (in fact, it would have worsened the problem). One of the issues in the failure is renewables, but not because of intermittence. It's because of their ~infinite ramp and them being DC. Anything that's not a spinning slug of steel produces AC through an inverter: electronics that take some DC, pass it through MOSFETs and coils, and spits out a mathematically pure sine wave on the output. They are perfectly controllable, and have no inertia: tell them tout output a set power and they happily will. However, this has a few specific issues: - infinite ramps produce sudden influx of energy or sudden drops in energy, which can trigger oscillations and trip safety of other plants - the sine wave being electronically generated, physics won't help you to keep it in phase with the network, and more crucially, keep it lagging/ahead of the network The last point is the most important one, and one that is actually discussed in the report. AC works well because physics is on our side, so spinning slugs or steel will self-correct depending on the power requirements of the grid, and this includes their phase compared to the grid. How out-of-phase you are is what's commonly called the power factor. Spinning slugs have a natural power factor, but inverter don't: you can make any power factor you want. Here in the spanish blackout, there was an excess of reactive power (that is, a phase shift happening). Spinning slugs will fight this shift of phase to realign with the correct phase. An inverter will happily follow the sine wave measured and contribute to the excess of reactive power. The report outlines this: there was no "market incentive" for inverters to actively correct the grid's power factor (trad: there are no fines). So really, more storage would not have helped. They would have tripped just like the other generators, and being inverter-based, they would have contributed to the issue. Not because "muh renewable" or "muh battery", but because of an inherent characteristic of how they're connected to the grid. Can this be fixed? Of course. We've had the technology for years for inverters to better mimic spinning slugs of steel. Will it be? Of course. Spain's TSO will make it a requirement to fix this and energy producers will comply. A few closing notes: - this is not an anti-renewables writeup, but an explanation of the tech, and the fact that renewables are part of the issue is a coincidence on the underlying technical details - inverters are not the reason the grid failed. but they're a part of why it had a runaway behavior - yes, wind also runs on inverters despite being spinning things. with the wind being so variable, it's much more efficient to have all turbines be not synchronized, convert their AC to DC, aggregate the DC, and convert back to AC when injecting into the grid
wortelefant 7mo ago
It is very carefully worded, but variable renewables are holding the smoking gun here. This is why spain now requests a better connection to french nuclear now. This reckless overbuild of variable generation is a valuable negative example, wind and solar without adequate hydro or nuclear is dead
- iririririr 7mo ago
  [dead]
- TheOtherHobbes 7mo ago
  It's lack of experience managing variability, not variability itself. Wind and solar are very far from dead, but they do need some adjustments - as the report makes clear.
- anthk 7mo ago
  Spaniard here, I didn't hear about that.
- Rygian 7mo ago
  Your statement is wrong. The report describes that there was no mechanism to dispatch the reactive power of renewables separately from the active power. In page 452, item numbered 1 states "RES power plants follow fixed power factor" (RES = Renewable Energy Sources). The source of this finding is in section 4.2.1. In page 208, footnote 35, the reference is given to Royal Decree 413/2014 of 6 June, which mandates this fixed power factor. The Article 7, section e), states that renewable energy sources must follow the instructions given by the operator to set power factor, and only if the distribution lines support it. And footnote 36 describes how this worked in practice on the date of the outage: renewables were told, by email on the previous day, which fixed power factor correction to use the following day. -- This lack of dynamic dispatch of reactive power was a known problem, already reported in 2022 [1] [1] https://www.eldiario.es/economia/competencia-reconocio-julio-2022-problemas-continuos-control-tension-red-electrica-peninsular_1_12299347.html https://www.eldiario.es/economia/competencia-reconocio-julio...

The fact that there is not a single root cause but several ones makes me instinctively think this is a good report, because it's not what the "bosses" (and even less politicians) like to hear.

ragebol 7mo ago

Yep, sounds like "This was bound to happen at some point"

cucumber3732842 7mo ago
Which on some level is exactly "what the bosses and politicians want to hear" When it's everybody's fault it's nobody's fault.
- drob518 7mo ago
  Exactly.
- darkwater 7mo ago
  In some ways, yes, but yet it's what reality is. There was probably some last factor kicking in that triggered the cascade, but there were probably many non-happy-paths not properly covered by working backup/fallback strategies. So a report could totally still tell "it's X fault", pointing the finger there. Government would blame the owner of X, some public statement about fixing X would be made and then the ones working in the field should internally push toi improve/fix their own (reduced) scope. I don't know what will come of this report in the next months/years, I will keep an eye on it though, since I live in Spain :)
- lyu07282 7mo ago
  But EU's liberalized energy market gives us resiliency and low prices for electricity! /s
  - galbar 7mo ago
    But not across the Pyrenees :_)

drob518 7mo ago

Frequently, when you see these massive failures, the root cause is an alignment of small weaknesses that all come together on a specific day. See, for instance, the space shuttle O-ring incident, Three-Mile Island, Fukushima, etc. These are complex systems with lots of moving parts and lots of (sometimes independent) people managing them. In a sense, the complexity it the common root cause.

amelius 7mo ago
It usually starts with a broken coffee machine.
- drob518 7mo ago
  When that happens, get ready.
linuxguy2 7mo ago
It's like the Swiss Cheese model where every system has "holes" or vulnerabilities, several layers, and a major incident only occurs when a hole aligns through all the layers. https://en.wikipedia.org/wiki/Swiss_cheese_model https://en.wikipedia.org/wiki/Swiss_cheese_model
- Ringz 7mo ago
  I use this model all the time. It's very helpful for explaining the multifactorial genesis of catastrophes to ordinary people.
  - anonymars 7mo ago
    Also perhaps worth a read: https://devblogs.microsoft.com/oldnewthing/20080416-00/?p=22723#:~:text=You%E2%80%99ve%20all%20experienced,errors%20are%20exposed. https://devblogs.microsoft.com/oldnewthing/20080416-00/?p=22... "You’ve all experienced the Fundamental Failure-Mode Theorem: You’re investigating a problem and along the way you find some function that never worked. A cache has a bug that results in cache misses when there should be hits. A request for an object that should be there somehow always fails. And yet the system still worked in spite of these errors. Eventually you trace the problem to a recent change that exposed all of the other bugs. Those bugs were always there, but the system kept on working because there was enough redundancy that one component was able to compensate for the failure of another component. Sometimes this chain of errors and compensation continues for several cycles, until finally the last protective layer fails and the underlying errors are exposed."

OgsyedIE 7mo ago

There are ways to aggregate these into a single resilience score for policy makers with only moderate loss of detail but it's unpopular.

red_admiral 7mo ago

Yes, a lot of modern engineering is good enough that single-cause failures are very rare indeed. That means that failures themselves are rare, but when they do happen, they're most likely to have multiple causes. How to explain that to non-engineers is another problem.

pseudohadamard 7mo ago
I think a better way of explaining it to people is that we've made critical systems so reliable that, in order for them to fail, the failures have to be quite complex.
javier2 7mo ago
we are not as complicated as the national grid, I have been here for nearly 10 years now, and our outages have gone from single cause, two causes, or now its nearly always 3 things that need to go wrong at the same time.
cameldrv 7mo ago
This is almost universal in aviation. They always talk about the "accident chain." Essentially everything that can kill you with one mistake is illegal through training and operational requirements and engineering and maintenance regulations.

toomuchtodo 7mo ago

They need more battery storage for grid health, both colocated at solar PV generators (to buffer voltage and frequency anomalies) and spread throughout the grid. This replaces inertia and other grid services provided by spinning thermal generators. There was no market mechanism to encourage the deployment of this technology in concert with Spain’s rapid deployment of solar and wind.

z2 7mo ago
There are non-battery buffers available too--I recently got rooftop residential solar installed, and learned that my area is covered by a grid profile requiring that the solar system stay online through something like 60 +/- 2Hz before shutting down completely, and ramping down production linearly beyond a 1Hz deviation or so. The point is to avoid cascading shutdowns by riding through over/undersupply situations, whereas an older standard for my area would have the all solar systems cut off the moment frequency exceeded 60.5Hz (which would indicate oversupply from power plant generators spinning faster via lower resistance). In my system's case, switching to this grid profile was just a software toggle.
- toomuchtodo 7mo ago
  This is grid following, very effective for small scale generation. It does not work for large scale generation though when the grid is relying on that voltage and frequency from the utility scale renewable generation ("grid forming"). When those large generators exceed their ride through tolerance, batteries step in to hold voltage and frequency up until the transient event ends or dispatchable generators called upon spin up (currently fossil gas primarily, but also nuclear if there is headroom to increase output). Thermal generators can take minutes to provide this support (called upon, fuel intake increased, spinning metal spins faster), batteries respond within 250-500ms. Tesla’s Megapack system at the Hornsdale Power Reserve in Australia was the first example of this being proven out at scale in prod. Batteries everywhere, as quickly as possible.
tuetuopay 7mo ago
In this very specific case, battery storage would not have helped (in fact, it would have worsened the problem). One of the issues in the failure is renewables, but not because of intermittence. It's because of their ~infinite ramp and them being DC. Anything that's not a spinning slug of steel produces AC through an inverter: electronics that take some DC, pass it through MOSFETs and coils, and spits out a mathematically pure sine wave on the output. They are perfectly controllable, and have no inertia: tell them tout output a set power and they happily will. However, this has a few specific issues: - infinite ramps produce sudden influx of energy or sudden drops in energy, which can trigger oscillations and trip safety of other plants - the sine wave being electronically generated, physics won't help you to keep it in phase with the network, and more crucially, keep it lagging/ahead of the network The last point is the most important one, and one that is actually discussed in the report. AC works well because physics is on our side, so spinning slugs or steel will self-correct depending on the power requirements of the grid, and this includes their phase compared to the grid. How out-of-phase you are is what's commonly called the power factor. Spinning slugs have a natural power factor, but inverter don't: you can make any power factor you want. Here in the spanish blackout, there was an excess of reactive power (that is, a phase shift happening). Spinning slugs will fight this shift of phase to realign with the correct phase. An inverter will happily follow the sine wave measured and contribute to the excess of reactive power. The report outlines this: there was no "market incentive" for inverters to actively correct the grid's power factor (trad: there are no fines). So really, more storage would not have helped. They would have tripped just like the other generators, and being inverter-based, they would have contributed to the issue. Not because "muh renewable" or "muh battery", but because of an inherent characteristic of how they're connected to the grid. Can this be fixed? Of course. We've had the technology for years for inverters to better mimic spinning slugs of steel. Will it be? Of course. Spain's TSO will make it a requirement to fix this and energy producers will comply. A few closing notes: - this is not an anti-renewables writeup, but an explanation of the tech, and the fact that renewables are part of the issue is a coincidence on the underlying technical details - inverters are not the reason the grid failed. but they're a part of why it had a runaway behavior - yes, wind also runs on inverters despite being spinning things. with the wind being so variable, it's much more efficient to have all turbines be not synchronized, convert their AC to DC, aggregate the DC, and convert back to AC when injecting into the grid

wortelefant 7mo ago

It is very carefully worded, but variable renewables are holding the smoking gun here. This is why spain now requests a better connection to french nuclear now. This reckless overbuild of variable generation is a valuable negative example, wind and solar without adequate hydro or nuclear is dead

iririririr 7mo ago
[dead]
TheOtherHobbes 7mo ago
It's lack of experience managing variability, not variability itself. Wind and solar are very far from dead, but they do need some adjustments - as the report makes clear.
anthk 7mo ago
Spaniard here, I didn't hear about that.
Rygian 7mo ago
Your statement is wrong. The report describes that there was no mechanism to dispatch the reactive power of renewables separately from the active power. In page 452, item numbered 1 states "RES power plants follow fixed power factor" (RES = Renewable Energy Sources). The source of this finding is in section 4.2.1. In page 208, footnote 35, the reference is given to Royal Decree 413/2014 of 6 June, which mandates this fixed power factor. The Article 7, section e), states that renewable energy sources must follow the instructions given by the operator to set power factor, and only if the distribution lines support it. And footnote 36 describes how this worked in practice on the date of the outage: renewables were told, by email on the previous day, which fixed power factor correction to use the following day. -- This lack of dynamic dispatch of reactive power was a known problem, already reported in 2022 [1] [1] https://www.eldiario.es/economia/competencia-reconocio-julio-2022-problemas-continuos-control-tension-red-electrica-peninsular_1_12299347.html https://www.eldiario.es/economia/competencia-reconocio-julio...