6 ms·

Entso-E final report on Iberian 2025 blackout

223 points by Rygian 7mo ago

deleted 7mo ago
[deleted]
darkwater 7mo ago
The fact that there is not a single root cause but several ones makes me instinctively think this is a good report, because it's not what the "bosses" (and even less politicians) like to hear.
- ragebol 7mo ago
  Yep, sounds like "This was bound to happen at some point"
  - cucumber3732842 7mo ago
    Which on some level is exactly "what the bosses and politicians want to hear" When it's everybody's fault it's nobody's fault.
    - drob518 7mo ago
      Exactly.
    - darkwater 7mo ago
      In some ways, yes, but yet it's what reality is. There was probably some last factor kicking in that triggered the cascade, but there were probably many non-happy-paths not properly covered by working backup/fallback strategies. So a report could totally still tell "it's X fault", pointing the finger there. Government would blame the owner of X, some public statement about fixing X would be made and then the ones working in the field should internally push toi improve/fix their own (reduced) scope. I don't know what will come of this report in the next months/years, I will keep an eye on it though, since I live in Spain :)
    - lyu07282 7mo ago
      But EU's liberalized energy market gives us resiliency and low prices for electricity! /s
      - galbar 7mo ago
        But not across the Pyrenees :_)
- drob518 7mo ago
  Frequently, when you see these massive failures, the root cause is an alignment of small weaknesses that all come together on a specific day. See, for instance, the space shuttle O-ring incident, Three-Mile Island, Fukushima, etc. These are complex systems with lots of moving parts and lots of (sometimes independent) people managing them. In a sense, the complexity it the common root cause.
  - amelius 7mo ago
    It usually starts with a broken coffee machine.
    - drob518 7mo ago
      When that happens, get ready.
  - linuxguy2 7mo ago
    It's like the Swiss Cheese model where every system has "holes" or vulnerabilities, several layers, and a major incident only occurs when a hole aligns through all the layers. https://en.wikipedia.org/wiki/Swiss_cheese_model https://en.wikipedia.org/wiki/Swiss_cheese_model
    - Ringz 7mo ago
      I use this model all the time. It's very helpful for explaining the multifactorial genesis of catastrophes to ordinary people.
      - anonymars 7mo ago
        Also perhaps worth a read: https://devblogs.microsoft.com/oldnewthing/20080416-00/?p=22723#:~:text=You%E2%80%99ve%20all%20experienced,errors%20are%20exposed. https://devblogs.microsoft.com/oldnewthing/20080416-00/?p=22... "You’ve all experienced the Fundamental Failure-Mode Theorem: You’re investigating a problem and along the way you find some function that never worked. A cache has a bug that results in cache misses when there should be hits. A request for an object that should be there somehow always fails. And yet the system still worked in spite of these errors. Eventually you trace the problem to a recent change that exposed all of the other bugs. Those bugs were always there, but the system kept on working because there was enough redundancy that one component was able to compensate for the failure of another component. Sometimes this chain of errors and compensation continues for several cycles, until finally the last protective layer fails and the underlying errors are exposed."
- OgsyedIE 7mo ago
  There are ways to aggregate these into a single resilience score for policy makers with only moderate loss of detail but it's unpopular.
- red_admiral 7mo ago
  Yes, a lot of modern engineering is good enough that single-cause failures are very rare indeed. That means that failures themselves are rare, but when they do happen, they're most likely to have multiple causes. How to explain that to non-engineers is another problem.
  - pseudohadamard 7mo ago
    I think a better way of explaining it to people is that we've made critical systems so reliable that, in order for them to fail, the failures have to be quite complex.
  - javier2 7mo ago
    we are not as complicated as the national grid, I have been here for nearly 10 years now, and our outages have gone from single cause, two causes, or now its nearly always 3 things that need to go wrong at the same time.
  - cameldrv 7mo ago
    This is almost universal in aviation. They always talk about the "accident chain." Essentially everything that can kill you with one mistake is illegal through training and operational requirements and engineering and maintenance regulations.
- toomuchtodo 7mo ago
  They need more battery storage for grid health, both colocated at solar PV generators (to buffer voltage and frequency anomalies) and spread throughout the grid. This replaces inertia and other grid services provided by spinning thermal generators. There was no market mechanism to encourage the deployment of this technology in concert with Spain’s rapid deployment of solar and wind.
  - z2 7mo ago
    There are non-battery buffers available too--I recently got rooftop residential solar installed, and learned that my area is covered by a grid profile requiring that the solar system stay online through something like 60 +/- 2Hz before shutting down completely, and ramping down production linearly beyond a 1Hz deviation or so. The point is to avoid cascading shutdowns by riding through over/undersupply situations, whereas an older standard for my area would have the all solar systems cut off the moment frequency exceeded 60.5Hz (which would indicate oversupply from power plant generators spinning faster via lower resistance). In my system's case, switching to this grid profile was just a software toggle.
    - toomuchtodo 7mo ago
      This is grid following, very effective for small scale generation. It does not work for large scale generation though when the grid is relying on that voltage and frequency from the utility scale renewable generation ("grid forming"). When those large generators exceed their ride through tolerance, batteries step in to hold voltage and frequency up until the transient event ends or dispatchable generators called upon spin up (currently fossil gas primarily, but also nuclear if there is headroom to increase output). Thermal generators can take minutes to provide this support (called upon, fuel intake increased, spinning metal spins faster), batteries respond within 250-500ms. Tesla’s Megapack system at the Hornsdale Power Reserve in Australia was the first example of this being proven out at scale in prod. Batteries everywhere, as quickly as possible.
  - tuetuopay 7mo ago
    In this very specific case, battery storage would not have helped (in fact, it would have worsened the problem). One of the issues in the failure is renewables, but not because of intermittence. It's because of their ~infinite ramp and them being DC. Anything that's not a spinning slug of steel produces AC through an inverter: electronics that take some DC, pass it through MOSFETs and coils, and spits out a mathematically pure sine wave on the output. They are perfectly controllable, and have no inertia: tell them tout output a set power and they happily will. However, this has a few specific issues: - infinite ramps produce sudden influx of energy or sudden drops in energy, which can trigger oscillations and trip safety of other plants - the sine wave being electronically generated, physics won't help you to keep it in phase with the network, and more crucially, keep it lagging/ahead of the network The last point is the most important one, and one that is actually discussed in the report. AC works well because physics is on our side, so spinning slugs or steel will self-correct depending on the power requirements of the grid, and this includes their phase compared to the grid. How out-of-phase you are is what's commonly called the power factor. Spinning slugs have a natural power factor, but inverter don't: you can make any power factor you want. Here in the spanish blackout, there was an excess of reactive power (that is, a phase shift happening). Spinning slugs will fight this shift of phase to realign with the correct phase. An inverter will happily follow the sine wave measured and contribute to the excess of reactive power. The report outlines this: there was no "market incentive" for inverters to actively correct the grid's power factor (trad: there are no fines). So really, more storage would not have helped. They would have tripped just like the other generators, and being inverter-based, they would have contributed to the issue. Not because "muh renewable" or "muh battery", but because of an inherent characteristic of how they're connected to the grid. Can this be fixed? Of course. We've had the technology for years for inverters to better mimic spinning slugs of steel. Will it be? Of course. Spain's TSO will make it a requirement to fix this and energy producers will comply. A few closing notes: - this is not an anti-renewables writeup, but an explanation of the tech, and the fact that renewables are part of the issue is a coincidence on the underlying technical details - inverters are not the reason the grid failed. but they're a part of why it had a runaway behavior - yes, wind also runs on inverters despite being spinning things. with the wind being so variable, it's much more efficient to have all turbines be not synchronized, convert their AC to DC, aggregate the DC, and convert back to AC when injecting into the grid
- wortelefant 7mo ago
  It is very carefully worded, but variable renewables are holding the smoking gun here. This is why spain now requests a better connection to french nuclear now. This reckless overbuild of variable generation is a valuable negative example, wind and solar without adequate hydro or nuclear is dead
  - iririririr 7mo ago
    [dead]
  - TheOtherHobbes 7mo ago
    It's lack of experience managing variability, not variability itself. Wind and solar are very far from dead, but they do need some adjustments - as the report makes clear.
  - anthk 7mo ago
    Spaniard here, I didn't hear about that.
  - Rygian 7mo ago
    Your statement is wrong. The report describes that there was no mechanism to dispatch the reactive power of renewables separately from the active power. In page 452, item numbered 1 states "RES power plants follow fixed power factor" (RES = Renewable Energy Sources). The source of this finding is in section 4.2.1. In page 208, footnote 35, the reference is given to Royal Decree 413/2014 of 6 June, which mandates this fixed power factor. The Article 7, section e), states that renewable energy sources must follow the instructions given by the operator to set power factor, and only if the distribution lines support it. And footnote 36 describes how this worked in practice on the date of the outage: renewables were told, by email on the previous day, which fixed power factor correction to use the following day. -- This lack of dynamic dispatch of reactive power was a known problem, already reported in 2022 [1] [1] https://www.eldiario.es/economia/competencia-reconocio-julio-2022-problemas-continuos-control-tension-red-electrica-peninsular_1_12299347.html https://www.eldiario.es/economia/competencia-reconocio-julio...
throw47452955 7mo ago
[dead]
algoth1 7mo ago
As someone who lived through the blackout it was wild. I felt back into the pre-internet, pre-smartphone era. It was pretty cool actually. The rumor mill spread so fast that Within hours the official word on the street was that we were getting hacked by a foreign military and people were joking that we had nothing of interest to be conquered xD
- pfortuny 7mo ago
  The hack thing spread wildly, indeed. Weird experience.
  - nunobrito 7mo ago
    In Germany a few months prior saw CCC publishing a method for destabilizing energy grids using radio waves a cheap hardware: https://media.ccc.de/v/38c3-blinkencity-radio-controlling-street-lamps-and-power-plants https://media.ccc.de/v/38c3-blinkencity-radio-controlling-st... and presented an attack vector to which most infrastructure in Europe is exposed. About 4 hours before the grid collapse on the 28th of April 2025 was recorded the largest purchase of Monero in the past 3 years (to remember: monero is coin of choice for special operations), making it surge +40% in 24 hours. The initial Spanish reports mentioned conflicting power information from dozens of locations at the same time which is consistent with a sequential attack using the blinkencity method so the grid itself is forced to close down.
    - rob74 7mo ago
      Well, if that's really the cause, then thanks CCC, I guess. For such a serious vulnerability which is probably non-trivial (not to mention expensive) to patch, is it really responsible to give only 3.5 months of time before disclosing it (according to slide #56 https://cdn.prod.website-files.com/5f6498c074436c349716e747/6776e858a69ec4927a9f4b5a_6776e76f3f601a2e318b8cae_blinkencity_slide56.png https://cdn.prod.website-files.com/5f6498c074436c349716e747/..., they notified EFR about the vulnerability on 2024-09-12 and disclosed it on 2024-12-28)?
      - 7mo ago
singhrac 7mo ago
I think people underestimate how valuable these reports are, so I’m very glad that detailed investigation is done here. Every major grid operator around the world is going to study this and make improvements to make sure this doesn’t happen on their grid. In a lot of ways it’s like investigations into airplane crashes.
- Rygian 7mo ago
  Obligatory read: https://how.complexsystems.fail/ https://how.complexsystems.fail/
- WJW 7mo ago
  The root cause tree on page 452 gives a good overview of how complex the behavior can be. The good news is that the grid operators have a good idea of what the problem was/is and it's well understood how to fix it. The downside is that it will require quite a bit of both time and money to reinforce the grid infrastructure.
- pseudohadamard 7mo ago
  Every major grid operator around the world is going to study this and make improvements to make sure this doesn’t happen on their grid. You mistyped "Every major grid operator is going to get their lawyers to reword their contracts to make sure they can't get sued when this happens".
  - pjc50 7mo ago
    Plenty of grids are publicly owned, or regulatory equivalent.
  - holgerschurig 7mo ago
    Spotted the US-american assuming their law system is used world-wide. If over here a lady buys a hot coffee in a McDrive, drives away, spills the hot coffee on their legs and makes a car accident due to this ... she won't be able to sue the McDrive. There's no fine-print or "Coffee is hot, you dumb person" writing needed anywhere. She could be lucky if she doesn't get fined for endangering others by her stupid actions. So, if we have a power outage here, the courts don't suddenly get busy. Because there simply no one is suing. Fun fact: despite this bad power outage, the power grid systems in Europe are still better (even way better) than in the US. There is a comparable statistics measure called "SAIDI" --- system average interuption duration index. And duration wise, per custom and year, the US power grids are worse than over here than in most of West Europe: (US SAIDI 2020: 1.3 hours, German SAIDI 2020: 0.3 hours). That's a factor of more than 4 on the worse-iness of US power grid! That could be an indicator that suing at the tiniest chance isn't helpful macro-ecnomical. Or that a general suing culture (with legalese trying to protect one from the economic risks) aren't actually helping improving things in the general sense, although they reduce the risk of getting bankrupt. But society-wise, a sue culture is most probably a negative: you spend energy/time/money on things that aren't necessary in saner law systems.
jacquesm 7mo ago
472 pages. That's going to be a nice bit of reading this weekend. It is very nice to see such a comprehensive report as well as the fact that it was made public immediately.
- dvh 7mo ago
  Maybe practical engineering will make a video about it
  - jacquesm 7mo ago
    I like reading.
    - dvh 7mo ago
      He also make article for each his videos, usually more technical than the video which tends to be more entrainment oriented.
wedg_ 7mo ago
I was supposed to fly home from Santiago de Compostella when the blackout happened. Me and my girlfriend had checked out of our hotel and headed to the bus stop to take the bus to the airport. The blackout had already started but we didn't realise (in hindsight, I do remember the pedestrian crossing not working. But I didn't think much of it). Anyways our flight was cancelled and it was clear we needed somewhere to stay the night. I immediately rebooked the same hotel, but when we got back there the receptionist had left so you had to check in over the phone instead. Except WhatsApp wasn't working. Then mobile data went down. And before long we were walking through the old town going hostel to hostel looking for a place to sleep, as everything got darker and darker (due to the lack of powered street lighting). The old town in almost pitch black was pretty scary! We ended up breaking back into the hotel, borrowing a bunch of towels from a laundry cart in the hallway and sleeping in this lockable room we found in the basement. Besides that somewhat stressful part, it was a really strange but fun experience to see the city without power: no traffic lights, darkened shops with lots of phone lights, cafés still operating just with only outdoor seating and limited menus, the occasional loud generator, and most of all the people seemingly having a great time in spite of it. I would've loved to have stayed out all night exploring the city, but finding somewhere to sleep that night was a bit more pressing!
AnotherGoodName 7mo ago
Can’t read all of this since it’s 424 pages but i want to point out that Australia is beating Europe on grid connected storage. Not on a per capita basis. It’s beating all of Europe combined outright https://www.visualcapitalist.com/top-20-countries-by-battery-storage-capacity/ https://www.visualcapitalist.com/top-20-countries-by-battery... We did have many many problems previously. The state of South Australia went out for a couple of weeks at one point in similar cascading failures. This doesn’t happen anymore. In fact the price of electricity is falling and the grid is more stable now https://www.theguardian.com/australia-news/2026/mar/19/power-bills-on-australias-east-coast-expected-to-fall-by-up-to-10-from-july-bringing-welcome-relief https://www.theguardian.com/australia-news/2026/mar/19/power... This price drop is inline with the lowered usage of gas turbine peaker plants (isn’t that helpful right now? No need for blockaded gas for electricity). A lot of people say it can’t be done. That you can’t have free power during the day (power is free on certain plans during daylight due to solar power inputs dropping wholesale prices to negative) and that you can’t build enough storage (still not there but the dent in gas turbine usage is clear). It’s one of these cases where you’ve been lied to. Australia elected a government that listened to reports battery+solar is great for grid reliability and nuclear was always going to be more expensive.
- postexitus 7mo ago
  You need grid connected storage where you have (unpredictable) renewables. That doesn't negate the benefits of Nuclear baseload power. In an ideal mix, you need both, and also Gas for emergencies. One is not better than the other, they have different roles in a balanced grid.
  - drtgh 7mo ago
    Plus, related (storage), you do not want to put hydroelectric in water reservoirs targeted to population consumption, as you could find out one summer that the reservoirs are empty, the result of such water being used with the intention of generate electricity, or even used as inertial stabilizer for renewables. This is the moment were at the news you read "There's a drought because it isn't raining" and similar excuses, when in reality your five years of water's reservoirs become reduced to half -or one third- due they focused the electricity production over the population real water demand. I mean, hydroelectric needs at least two level’s reservoirs, one to generate electricity (or even exclusive two level's reservoirs with water pumps for this), and the next one, absolutely untouchable by the electric companies, targeted as water storage for the population/agriculture, the classic more than five years reservoir, for real.
deleted 7mo ago
[deleted]
NooneAtAll3 7mo ago
If someone wants a "quick and dirty" answers - there's presentation linked https://eepublicdownloads.blob.core.windows.net/public-cdn-container/clean-documents/Publications/2025/iberian-blackout/260318_Final%20report_press%20briefing%20PPT_Final.pdf https://eepublicdownloads.blob.core.windows.net/public-cdn-c... page 11 contains "Full root cause tree" - one image with all the high level info
mythern 7mo ago
That was quite the interesting read!
wedge01 7mo ago
0.63 Hz and 0.2 Hz grid instability. Oh my.
christkv 7mo ago
Lol it took about an hour for us to realize there was a blackout as our house had switched to island mode. Ended being the sort of organization point of the area for the neighborhood as we could provide mobile charging and light until the power came back around 22 at night
- chr15m 7mo ago
  That's epic, congratulations on your foresight!
jefftrebben 7mo ago
[dead]
anthk 7mo ago
In my case I was near and Bilbao the blackout ended around ~12PM while other towns around -up to 12km far- didn't got the power back until 22PM. Everyone was outside with pocket radio receivers listening to instructions and having a good time around bars as if it were a European soccer final cup with the local team being playing... On telecomms, I barely got 2G speeds and no Spanish newspaper (nor media) dared to set a https://lite.cnn.com https://lite.cnn.com or https://text.npr.org https://text.npr.org like page to read the news. I had to literaly use a Gemini (the protocol) client, Lagrange, tunneling the webs thru gemini://gemi.dev with their 'News Waffle' service. And for international news I just resorted to Gopher and magical.fish. The mainstream web barely loaded some of the text (if any), while several IM options ran at crawling network speeds. The lines were clogged and I coudn't even call my relatives.

223 points by Rygian 7mo ago

deleted 7mo ago

[deleted]

The fact that there is not a single root cause but several ones makes me instinctively think this is a good report, because it's not what the "bosses" (and even less politicians) like to hear.

ragebol 7mo ago
Yep, sounds like "This was bound to happen at some point"
- cucumber3732842 7mo ago
  Which on some level is exactly "what the bosses and politicians want to hear" When it's everybody's fault it's nobody's fault.
  - drob518 7mo ago
    Exactly.
  - darkwater 7mo ago
    In some ways, yes, but yet it's what reality is. There was probably some last factor kicking in that triggered the cascade, but there were probably many non-happy-paths not properly covered by working backup/fallback strategies. So a report could totally still tell "it's X fault", pointing the finger there. Government would blame the owner of X, some public statement about fixing X would be made and then the ones working in the field should internally push toi improve/fix their own (reduced) scope. I don't know what will come of this report in the next months/years, I will keep an eye on it though, since I live in Spain :)
  - lyu07282 7mo ago
    But EU's liberalized energy market gives us resiliency and low prices for electricity! /s
    - galbar 7mo ago
      But not across the Pyrenees :_)
drob518 7mo ago
Frequently, when you see these massive failures, the root cause is an alignment of small weaknesses that all come together on a specific day. See, for instance, the space shuttle O-ring incident, Three-Mile Island, Fukushima, etc. These are complex systems with lots of moving parts and lots of (sometimes independent) people managing them. In a sense, the complexity it the common root cause.
- amelius 7mo ago
  It usually starts with a broken coffee machine.
  - drob518 7mo ago
    When that happens, get ready.
- linuxguy2 7mo ago
  It's like the Swiss Cheese model where every system has "holes" or vulnerabilities, several layers, and a major incident only occurs when a hole aligns through all the layers. https://en.wikipedia.org/wiki/Swiss_cheese_model https://en.wikipedia.org/wiki/Swiss_cheese_model
  - Ringz 7mo ago
    I use this model all the time. It's very helpful for explaining the multifactorial genesis of catastrophes to ordinary people.
    - anonymars 7mo ago
      Also perhaps worth a read: https://devblogs.microsoft.com/oldnewthing/20080416-00/?p=22723#:~:text=You%E2%80%99ve%20all%20experienced,errors%20are%20exposed. https://devblogs.microsoft.com/oldnewthing/20080416-00/?p=22... "You’ve all experienced the Fundamental Failure-Mode Theorem: You’re investigating a problem and along the way you find some function that never worked. A cache has a bug that results in cache misses when there should be hits. A request for an object that should be there somehow always fails. And yet the system still worked in spite of these errors. Eventually you trace the problem to a recent change that exposed all of the other bugs. Those bugs were always there, but the system kept on working because there was enough redundancy that one component was able to compensate for the failure of another component. Sometimes this chain of errors and compensation continues for several cycles, until finally the last protective layer fails and the underlying errors are exposed."
OgsyedIE 7mo ago
There are ways to aggregate these into a single resilience score for policy makers with only moderate loss of detail but it's unpopular.
red_admiral 7mo ago
Yes, a lot of modern engineering is good enough that single-cause failures are very rare indeed. That means that failures themselves are rare, but when they do happen, they're most likely to have multiple causes. How to explain that to non-engineers is another problem.
- pseudohadamard 7mo ago
  I think a better way of explaining it to people is that we've made critical systems so reliable that, in order for them to fail, the failures have to be quite complex.
- javier2 7mo ago
  we are not as complicated as the national grid, I have been here for nearly 10 years now, and our outages have gone from single cause, two causes, or now its nearly always 3 things that need to go wrong at the same time.
- cameldrv 7mo ago
  This is almost universal in aviation. They always talk about the "accident chain." Essentially everything that can kill you with one mistake is illegal through training and operational requirements and engineering and maintenance regulations.
toomuchtodo 7mo ago
They need more battery storage for grid health, both colocated at solar PV generators (to buffer voltage and frequency anomalies) and spread throughout the grid. This replaces inertia and other grid services provided by spinning thermal generators. There was no market mechanism to encourage the deployment of this technology in concert with Spain’s rapid deployment of solar and wind.
- z2 7mo ago
  There are non-battery buffers available too--I recently got rooftop residential solar installed, and learned that my area is covered by a grid profile requiring that the solar system stay online through something like 60 +/- 2Hz before shutting down completely, and ramping down production linearly beyond a 1Hz deviation or so. The point is to avoid cascading shutdowns by riding through over/undersupply situations, whereas an older standard for my area would have the all solar systems cut off the moment frequency exceeded 60.5Hz (which would indicate oversupply from power plant generators spinning faster via lower resistance). In my system's case, switching to this grid profile was just a software toggle.
  - toomuchtodo 7mo ago
    This is grid following, very effective for small scale generation. It does not work for large scale generation though when the grid is relying on that voltage and frequency from the utility scale renewable generation ("grid forming"). When those large generators exceed their ride through tolerance, batteries step in to hold voltage and frequency up until the transient event ends or dispatchable generators called upon spin up (currently fossil gas primarily, but also nuclear if there is headroom to increase output). Thermal generators can take minutes to provide this support (called upon, fuel intake increased, spinning metal spins faster), batteries respond within 250-500ms. Tesla’s Megapack system at the Hornsdale Power Reserve in Australia was the first example of this being proven out at scale in prod. Batteries everywhere, as quickly as possible.
- tuetuopay 7mo ago
  In this very specific case, battery storage would not have helped (in fact, it would have worsened the problem). One of the issues in the failure is renewables, but not because of intermittence. It's because of their ~infinite ramp and them being DC. Anything that's not a spinning slug of steel produces AC through an inverter: electronics that take some DC, pass it through MOSFETs and coils, and spits out a mathematically pure sine wave on the output. They are perfectly controllable, and have no inertia: tell them tout output a set power and they happily will. However, this has a few specific issues: - infinite ramps produce sudden influx of energy or sudden drops in energy, which can trigger oscillations and trip safety of other plants - the sine wave being electronically generated, physics won't help you to keep it in phase with the network, and more crucially, keep it lagging/ahead of the network The last point is the most important one, and one that is actually discussed in the report. AC works well because physics is on our side, so spinning slugs or steel will self-correct depending on the power requirements of the grid, and this includes their phase compared to the grid. How out-of-phase you are is what's commonly called the power factor. Spinning slugs have a natural power factor, but inverter don't: you can make any power factor you want. Here in the spanish blackout, there was an excess of reactive power (that is, a phase shift happening). Spinning slugs will fight this shift of phase to realign with the correct phase. An inverter will happily follow the sine wave measured and contribute to the excess of reactive power. The report outlines this: there was no "market incentive" for inverters to actively correct the grid's power factor (trad: there are no fines). So really, more storage would not have helped. They would have tripped just like the other generators, and being inverter-based, they would have contributed to the issue. Not because "muh renewable" or "muh battery", but because of an inherent characteristic of how they're connected to the grid. Can this be fixed? Of course. We've had the technology for years for inverters to better mimic spinning slugs of steel. Will it be? Of course. Spain's TSO will make it a requirement to fix this and energy producers will comply. A few closing notes: - this is not an anti-renewables writeup, but an explanation of the tech, and the fact that renewables are part of the issue is a coincidence on the underlying technical details - inverters are not the reason the grid failed. but they're a part of why it had a runaway behavior - yes, wind also runs on inverters despite being spinning things. with the wind being so variable, it's much more efficient to have all turbines be not synchronized, convert their AC to DC, aggregate the DC, and convert back to AC when injecting into the grid
wortelefant 7mo ago
It is very carefully worded, but variable renewables are holding the smoking gun here. This is why spain now requests a better connection to french nuclear now. This reckless overbuild of variable generation is a valuable negative example, wind and solar without adequate hydro or nuclear is dead
- iririririr 7mo ago
  [dead]
- TheOtherHobbes 7mo ago
  It's lack of experience managing variability, not variability itself. Wind and solar are very far from dead, but they do need some adjustments - as the report makes clear.
- anthk 7mo ago
  Spaniard here, I didn't hear about that.
- Rygian 7mo ago
  Your statement is wrong. The report describes that there was no mechanism to dispatch the reactive power of renewables separately from the active power. In page 452, item numbered 1 states "RES power plants follow fixed power factor" (RES = Renewable Energy Sources). The source of this finding is in section 4.2.1. In page 208, footnote 35, the reference is given to Royal Decree 413/2014 of 6 June, which mandates this fixed power factor. The Article 7, section e), states that renewable energy sources must follow the instructions given by the operator to set power factor, and only if the distribution lines support it. And footnote 36 describes how this worked in practice on the date of the outage: renewables were told, by email on the previous day, which fixed power factor correction to use the following day. -- This lack of dynamic dispatch of reactive power was a known problem, already reported in 2022 [1] [1] https://www.eldiario.es/economia/competencia-reconocio-julio-2022-problemas-continuos-control-tension-red-electrica-peninsular_1_12299347.html https://www.eldiario.es/economia/competencia-reconocio-julio...

throw47452955 7mo ago

[dead]

algoth1 7mo ago

As someone who lived through the blackout it was wild. I felt back into the pre-internet, pre-smartphone era. It was pretty cool actually. The rumor mill spread so fast that Within hours the official word on the street was that we were getting hacked by a foreign military and people were joking that we had nothing of interest to be conquered xD

pfortuny 7mo ago
The hack thing spread wildly, indeed. Weird experience.
- nunobrito 7mo ago
  In Germany a few months prior saw CCC publishing a method for destabilizing energy grids using radio waves a cheap hardware: https://media.ccc.de/v/38c3-blinkencity-radio-controlling-street-lamps-and-power-plants https://media.ccc.de/v/38c3-blinkencity-radio-controlling-st... and presented an attack vector to which most infrastructure in Europe is exposed. About 4 hours before the grid collapse on the 28th of April 2025 was recorded the largest purchase of Monero in the past 3 years (to remember: monero is coin of choice for special operations), making it surge +40% in 24 hours. The initial Spanish reports mentioned conflicting power information from dozens of locations at the same time which is consistent with a sequential attack using the blinkencity method so the grid itself is forced to close down.
  - rob74 7mo ago
    Well, if that's really the cause, then thanks CCC, I guess. For such a serious vulnerability which is probably non-trivial (not to mention expensive) to patch, is it really responsible to give only 3.5 months of time before disclosing it (according to slide #56 https://cdn.prod.website-files.com/5f6498c074436c349716e747/6776e858a69ec4927a9f4b5a_6776e76f3f601a2e318b8cae_blinkencity_slide56.png https://cdn.prod.website-files.com/5f6498c074436c349716e747/..., they notified EFR about the vulnerability on 2024-09-12 and disclosed it on 2024-12-28)?
    - 7mo ago

singhrac 7mo ago

I think people underestimate how valuable these reports are, so I’m very glad that detailed investigation is done here. Every major grid operator around the world is going to study this and make improvements to make sure this doesn’t happen on their grid. In a lot of ways it’s like investigations into airplane crashes.

Rygian 7mo ago
Obligatory read: https://how.complexsystems.fail/ https://how.complexsystems.fail/
WJW 7mo ago
The root cause tree on page 452 gives a good overview of how complex the behavior can be. The good news is that the grid operators have a good idea of what the problem was/is and it's well understood how to fix it. The downside is that it will require quite a bit of both time and money to reinforce the grid infrastructure.
pseudohadamard 7mo ago
Every major grid operator around the world is going to study this and make improvements to make sure this doesn’t happen on their grid. You mistyped "Every major grid operator is going to get their lawyers to reword their contracts to make sure they can't get sued when this happens".
- pjc50 7mo ago
  Plenty of grids are publicly owned, or regulatory equivalent.
- holgerschurig 7mo ago
  Spotted the US-american assuming their law system is used world-wide. If over here a lady buys a hot coffee in a McDrive, drives away, spills the hot coffee on their legs and makes a car accident due to this ... she won't be able to sue the McDrive. There's no fine-print or "Coffee is hot, you dumb person" writing needed anywhere. She could be lucky if she doesn't get fined for endangering others by her stupid actions. So, if we have a power outage here, the courts don't suddenly get busy. Because there simply no one is suing. Fun fact: despite this bad power outage, the power grid systems in Europe are still better (even way better) than in the US. There is a comparable statistics measure called "SAIDI" --- system average interuption duration index. And duration wise, per custom and year, the US power grids are worse than over here than in most of West Europe: (US SAIDI 2020: 1.3 hours, German SAIDI 2020: 0.3 hours). That's a factor of more than 4 on the worse-iness of US power grid! That could be an indicator that suing at the tiniest chance isn't helpful macro-ecnomical. Or that a general suing culture (with legalese trying to protect one from the economic risks) aren't actually helping improving things in the general sense, although they reduce the risk of getting bankrupt. But society-wise, a sue culture is most probably a negative: you spend energy/time/money on things that aren't necessary in saner law systems.

jacquesm 7mo ago

472 pages. That's going to be a nice bit of reading this weekend. It is very nice to see such a comprehensive report as well as the fact that it was made public immediately.

dvh 7mo ago
Maybe practical engineering will make a video about it
- jacquesm 7mo ago
  I like reading.
  - dvh 7mo ago
    He also make article for each his videos, usually more technical than the video which tends to be more entrainment oriented.

wedg_ 7mo ago

I was supposed to fly home from Santiago de Compostella when the blackout happened. Me and my girlfriend had checked out of our hotel and headed to the bus stop to take the bus to the airport. The blackout had already started but we didn't realise (in hindsight, I do remember the pedestrian crossing not working. But I didn't think much of it). Anyways our flight was cancelled and it was clear we needed somewhere to stay the night. I immediately rebooked the same hotel, but when we got back there the receptionist had left so you had to check in over the phone instead. Except WhatsApp wasn't working. Then mobile data went down. And before long we were walking through the old town going hostel to hostel looking for a place to sleep, as everything got darker and darker (due to the lack of powered street lighting). The old town in almost pitch black was pretty scary! We ended up breaking back into the hotel, borrowing a bunch of towels from a laundry cart in the hallway and sleeping in this lockable room we found in the basement. Besides that somewhat stressful part, it was a really strange but fun experience to see the city without power: no traffic lights, darkened shops with lots of phone lights, cafés still operating just with only outdoor seating and limited menus, the occasional loud generator, and most of all the people seemingly having a great time in spite of it. I would've loved to have stayed out all night exploring the city, but finding somewhere to sleep that night was a bit more pressing!

AnotherGoodName 7mo ago

Can’t read all of this since it’s 424 pages but i want to point out that Australia is beating Europe on grid connected storage. Not on a per capita basis. It’s beating all of Europe combined outright https://www.visualcapitalist.com/top-20-countries-by-battery-storage-capacity/ https://www.visualcapitalist.com/top-20-countries-by-battery... We did have many many problems previously. The state of South Australia went out for a couple of weeks at one point in similar cascading failures. This doesn’t happen anymore. In fact the price of electricity is falling and the grid is more stable now https://www.theguardian.com/australia-news/2026/mar/19/power-bills-on-australias-east-coast-expected-to-fall-by-up-to-10-from-july-bringing-welcome-relief https://www.theguardian.com/australia-news/2026/mar/19/power... This price drop is inline with the lowered usage of gas turbine peaker plants (isn’t that helpful right now? No need for blockaded gas for electricity). A lot of people say it can’t be done. That you can’t have free power during the day (power is free on certain plans during daylight due to solar power inputs dropping wholesale prices to negative) and that you can’t build enough storage (still not there but the dent in gas turbine usage is clear). It’s one of these cases where you’ve been lied to. Australia elected a government that listened to reports battery+solar is great for grid reliability and nuclear was always going to be more expensive.

postexitus 7mo ago
You need grid connected storage where you have (unpredictable) renewables. That doesn't negate the benefits of Nuclear baseload power. In an ideal mix, you need both, and also Gas for emergencies. One is not better than the other, they have different roles in a balanced grid.
- drtgh 7mo ago
  Plus, related (storage), you do not want to put hydroelectric in water reservoirs targeted to population consumption, as you could find out one summer that the reservoirs are empty, the result of such water being used with the intention of generate electricity, or even used as inertial stabilizer for renewables. This is the moment were at the news you read "There's a drought because it isn't raining" and similar excuses, when in reality your five years of water's reservoirs become reduced to half -or one third- due they focused the electricity production over the population real water demand. I mean, hydroelectric needs at least two level’s reservoirs, one to generate electricity (or even exclusive two level's reservoirs with water pumps for this), and the next one, absolutely untouchable by the electric companies, targeted as water storage for the population/agriculture, the classic more than five years reservoir, for real.

deleted 7mo ago

[deleted]

NooneAtAll3 7mo ago

If someone wants a "quick and dirty" answers - there's presentation linked https://eepublicdownloads.blob.core.windows.net/public-cdn-container/clean-documents/Publications/2025/iberian-blackout/260318_Final%20report_press%20briefing%20PPT_Final.pdf https://eepublicdownloads.blob.core.windows.net/public-cdn-c... page 11 contains "Full root cause tree" - one image with all the high level info

mythern 7mo ago

That was quite the interesting read!

wedge01 7mo ago

0.63 Hz and 0.2 Hz grid instability. Oh my.

christkv 7mo ago

Lol it took about an hour for us to realize there was a blackout as our house had switched to island mode. Ended being the sort of organization point of the area for the neighborhood as we could provide mobile charging and light until the power came back around 22 at night

chr15m 7mo ago
That's epic, congratulations on your foresight!

jefftrebben 7mo ago

[dead]

anthk 7mo ago

In my case I was near and Bilbao the blackout ended around ~12PM while other towns around -up to 12km far- didn't got the power back until 22PM. Everyone was outside with pocket radio receivers listening to instructions and having a good time around bars as if it were a European soccer final cup with the local team being playing... On telecomms, I barely got 2G speeds and no Spanish newspaper (nor media) dared to set a https://lite.cnn.com https://lite.cnn.com or https://text.npr.org https://text.npr.org like page to read the news. I had to literaly use a Gemini (the protocol) client, Lagrange, tunneling the webs thru gemini://gemi.dev with their 'News Waffle' service. And for international news I just resorted to Gopher and magical.fish. The mainstream web barely loaded some of the text (if any), while several IM options ran at crawling network speeds. The lines were clogged and I coudn't even call my relatives.