4 ms·

Miscalculation by Spanish power grid operator REE contributed to blackout

129 points by croes 1y ago

tux3 1y ago
They've now presented the report, but the report itself hasn't yet been made public. The press release is here: https://www.miteco.gob.es/es/prensa/ultimas-noticias/2025/junio/se-presenta-el-informe-del-comite-de-analisis-de-la-crisis-elect.html https://www.miteco.gob.es/es/prensa/ultimas-noticias/2025/ju...
- kragen 1y ago
  That's fantastic information, thanks! When will the report be made public?
  - diggan 1y ago
    Last I heard they were aiming for sometime between July and October to have the full technical report ready and shared with the parlament and the EU commission.
  - tux3 1y ago
    My understanding is that it was supposed to be released later today. It's 8PM in Spain right now.
- riedel 1y ago
  Interesting read. I reminded me of some stories a German guy who used to work for a Spanish utility told me (the guy lost his job at one of the first closed German nuclear facilities and now worked at the museum of a pumped storage power plant): he told us that the Spanish utility he worked for waited far too long to kick in gas power plants waiting for prices to rise further making tons of money by risking stability. I also learned from him that a lot German nuclear facilities are still running there generators only as weight to stabilize the network. (Actually he also told us that some utilities legally abuse the pumped storage to trade energy between networks). So it seems there are some strange incentives and necessities in today's system , that might explain the failures to react in time.
  - mschuster91 1y ago
    > I also learned from him that a lot German nuclear facilities are still running there generators only as weight to stabilize the network. We aren't. Pretty much the first things that get dismantled is the turbines and machinery that has no radiation exposure because you need space to maneuver, buffer, dismantle and decontaminate everything else that's coming out of the innards of the plant and actually is contaminated. Maybe you can get some remaining use as reactive power provider done during the ~1 year that it takes for the bureaucracy to process the actual permit for dismantling [1] - but even during that time, restarting it again is practically impossible after the main coolant pipes have been flushed [2] (that happened shortly after the plant had been formally taken offline and all the uranium pellets were removed from the core, although I can't find out how many months that took - but it did happen in the same year as the shutdown, so less than 8 months). The only thing that's being kept connected to the grid after that is the small portion that supplies the cooling for the cooldown pools where the freshly removed uranium pellets deplete enough material that they can be stored in CASTOR containers [3]. In addition, the high-voltage switching equipment is being kept alive in Landshut's case as the area of the former NPP is planned to be the end point of the SüdOstLink national grid expansion link and the power is supposed to be transferred from Ohu to the wider regional/state grid using the existing tie-ins of the former NPP [4]. If you want I can try to contact the utility handling the teardown about specifics, I live a few kilometers from there after all, but I doubt they're going to share much information given the obvious threat of Russian espionage. It's also a workplace safety issue. You absolutely do not want electricity anywhere where you do not vitally need it, particularly no high voltage. EDIT: Thanks to @closewith, indeed Biblis A's genset was converted for a "second life" as power regulator in 2011/2012 [5], but that's (at least from a cursory research) all I could find. [1] https://www.br.de/nachrichten/bayern/genehmigung-fuer-rueckbau-des-atomkraftwerks-isar-2-erteilt,U7kkOt0 https://www.br.de/nachrichten/bayern/genehmigung-fuer-rueckb... [2] https://www.bayerische-staatszeitung.de/staatszeitung/politik/detailansicht-politik/artikel/rueckbau-von-bayerns-atommeilern-geht-schnell-voran.html#topPosition https://www.bayerische-staatszeitung.de/staatszeitung/politi... [3] https://de.wikipedia.org/wiki/Castor_(Kerntechnik) https://de.wikipedia.org/wiki/Castor_(Kerntechnik) [4] https://www.br.de/nachrichten/bayern/tennet-stellt-suedostlink-trasse-von-pfatter-nach-landshut-vor,Tbs9hdI https://www.br.de/nachrichten/bayern/tennet-stellt-suedostli... [5] https://www.amprion.net/Presse/Presse-Detailseite_2667.html https://www.amprion.net/Presse/Presse-Detailseite_2667.html
narag 1y ago
Disclaimer: I can't really vouch for any of this, but let me tell you what's being discussed in the press over here. This is Government's version and it shows. REE boss was put there by the ruling party. Still they try to paint REE as a private operator now. It's not. Like many other orgs, it's a mix of private and public capital, ultimately controled by the government. It's suspected that the blackout was caused because she overruled technicians' opinion, looking for a solar % record. Not "enough thermal power stations" carefully avoids mentioning nuclear, that they want to erradicate. Private companies are requesting the records of CECOEL conversation to be published, some have leaked their logs. In one of those, REE technicians readily admit that oscilations are caused by not enough nuclear in the mix.
- deleted 1y ago
  [deleted]
- toomuchtodo 1y ago
  Doesn't have to be nuclear, can be anything that provides reactive power and grid services, i.e. grid forming battery storage. The problem is this service is needed on the Spanish grid, but isn't paid for in the market. Pay for it and the grid services will be provided. Resources below. Mind you, doesn't have to be a Tesla battery, just has to be a battery with power controls (grid forming) and some lithium cells for fast response (vs sodium for longer duration discharge). China launches world’s first grid-forming sodium-ion battery storage plant - https://www.ess-news.com/2025/06/03/china-launches-worlds-first-grid-forming-sodium-ion-battery-storage-plant/ https://www.ess-news.com/2025/06/03/china-launches-worlds-fi... - June 3rd, 2025 ("With a total investment of over CNY 460 million [$63.8 million] and occupying 34k square metres, the Baochi plant is designed for an installed capacity of 200MW/400MWh. Based on a dual daily charge-discharge cycle, it can regulate up to 580 GWh annually — enough to power 270,000 households, with 98 per cent of its energy sourced from renewables. The facility supports more than 30 local wind and solar power stations, alleviating the impact of intermittent supply and facilitating the integration of high shares of renewables into the grid.") Tesla Megapack in Texas Supports the Grid and Keeps the Lights On [Gambit Energy Storage] - https://www.tesla.com/videos/gambit-megapack https://www.tesla.com/videos/gambit-megapack - September 30, 2021 Tesla’s giant battery in Australia [Hornsdale Power Reserve] reduced grid service cost by 90% - https://news.ycombinator.com/item?id=17051066 https://news.ycombinator.com/item?id=17051066 - May 2018 Tesla Powerpacks Balancing the Grid in Terhills, Belgium [video] - https://www.youtube.com/watch?v=EVHQFrGzThg https://www.youtube.com/watch?v=EVHQFrGzThg - May 14, 2018
ZeroGravitas 1y ago
I think a gas plant lying about it's availability was eventually discovered to be a root cause of the South Australia blackout a few years back.
Rygian 1y ago
For me the keywords are "capacidad de control de tensión insuficiente" (insufficient voltage control capacity) which was already a known issue for years. REE asked back in 2019/2020 to create a new real-time dispatch mechanism for reactive power, because Spain still relies on fixed dispatch based on emails, phone calls, even snail mail. There was a pilot project in 2022 which was successful (dispatching in less than 5 seconds) but proved to be very expensive, and therefore incompatible with the European Union's directives on creating market-ready regulation processes [1]. [1][ES] https://www.eldiario.es/economia/competencia-reconocio-julio-2022-problemas-continuos-control-tension-red-electrica-peninsular_1_12299347.html https://www.eldiario.es/economia/competencia-reconocio-julio...
- UltraSane 1y ago
  "Spain still relies on fixed dispatch based on emails, phone calls, even snail mail. " Yikes!
demarq 1y ago
Wasn’t it supposed to be Russia
- b3orn 1y ago
  No, it was supposedly induced atmospheric vibration.
Dr_Knows 1y ago
@narag excellent recap. "carefully avoids mentioning nuclear" -- the nuclear operator/s decided not to supply electricitiy on that day.
cryptonector 1y ago
> REE did not have enough thermal power stations switched on when surge occurred That's.. that's a miscalculation alright. But it's not likely that it was just a small math error. It must have been policy to live on the edge.
- Rygian 1y ago
  It's common policy to plan your resource consumption in the most economical way. Couple that to a "next day" dispatching system. Couple that to resources that failed to do their job (did not compensate enough reactive power, or even added extra reactive power instead of compensating). Couple that to one of the most challenging electrical grids in the world. When taking that context in consideration, "living in the edge" is not really a legitimate way to put it.
mpweiher 1y ago
"REE did not have enough thermal power stations switched on when surge occurred"

129 points by croes 1y ago

tux3 1y ago

They've now presented the report, but the report itself hasn't yet been made public. The press release is here: https://www.miteco.gob.es/es/prensa/ultimas-noticias/2025/junio/se-presenta-el-informe-del-comite-de-analisis-de-la-crisis-elect.html https://www.miteco.gob.es/es/prensa/ultimas-noticias/2025/ju...

kragen 1y ago
That's fantastic information, thanks! When will the report be made public?
- diggan 1y ago
  Last I heard they were aiming for sometime between July and October to have the full technical report ready and shared with the parlament and the EU commission.
- tux3 1y ago
  My understanding is that it was supposed to be released later today. It's 8PM in Spain right now.
riedel 1y ago
Interesting read. I reminded me of some stories a German guy who used to work for a Spanish utility told me (the guy lost his job at one of the first closed German nuclear facilities and now worked at the museum of a pumped storage power plant): he told us that the Spanish utility he worked for waited far too long to kick in gas power plants waiting for prices to rise further making tons of money by risking stability. I also learned from him that a lot German nuclear facilities are still running there generators only as weight to stabilize the network. (Actually he also told us that some utilities legally abuse the pumped storage to trade energy between networks). So it seems there are some strange incentives and necessities in today's system , that might explain the failures to react in time.
- mschuster91 1y ago
  > I also learned from him that a lot German nuclear facilities are still running there generators only as weight to stabilize the network. We aren't. Pretty much the first things that get dismantled is the turbines and machinery that has no radiation exposure because you need space to maneuver, buffer, dismantle and decontaminate everything else that's coming out of the innards of the plant and actually is contaminated. Maybe you can get some remaining use as reactive power provider done during the ~1 year that it takes for the bureaucracy to process the actual permit for dismantling [1] - but even during that time, restarting it again is practically impossible after the main coolant pipes have been flushed [2] (that happened shortly after the plant had been formally taken offline and all the uranium pellets were removed from the core, although I can't find out how many months that took - but it did happen in the same year as the shutdown, so less than 8 months). The only thing that's being kept connected to the grid after that is the small portion that supplies the cooling for the cooldown pools where the freshly removed uranium pellets deplete enough material that they can be stored in CASTOR containers [3]. In addition, the high-voltage switching equipment is being kept alive in Landshut's case as the area of the former NPP is planned to be the end point of the SüdOstLink national grid expansion link and the power is supposed to be transferred from Ohu to the wider regional/state grid using the existing tie-ins of the former NPP [4]. If you want I can try to contact the utility handling the teardown about specifics, I live a few kilometers from there after all, but I doubt they're going to share much information given the obvious threat of Russian espionage. It's also a workplace safety issue. You absolutely do not want electricity anywhere where you do not vitally need it, particularly no high voltage. EDIT: Thanks to @closewith, indeed Biblis A's genset was converted for a "second life" as power regulator in 2011/2012 [5], but that's (at least from a cursory research) all I could find. [1] https://www.br.de/nachrichten/bayern/genehmigung-fuer-rueckbau-des-atomkraftwerks-isar-2-erteilt,U7kkOt0 https://www.br.de/nachrichten/bayern/genehmigung-fuer-rueckb... [2] https://www.bayerische-staatszeitung.de/staatszeitung/politik/detailansicht-politik/artikel/rueckbau-von-bayerns-atommeilern-geht-schnell-voran.html#topPosition https://www.bayerische-staatszeitung.de/staatszeitung/politi... [3] https://de.wikipedia.org/wiki/Castor_(Kerntechnik) https://de.wikipedia.org/wiki/Castor_(Kerntechnik) [4] https://www.br.de/nachrichten/bayern/tennet-stellt-suedostlink-trasse-von-pfatter-nach-landshut-vor,Tbs9hdI https://www.br.de/nachrichten/bayern/tennet-stellt-suedostli... [5] https://www.amprion.net/Presse/Presse-Detailseite_2667.html https://www.amprion.net/Presse/Presse-Detailseite_2667.html

narag 1y ago

Disclaimer: I can't really vouch for any of this, but let me tell you what's being discussed in the press over here. This is Government's version and it shows. REE boss was put there by the ruling party. Still they try to paint REE as a private operator now. It's not. Like many other orgs, it's a mix of private and public capital, ultimately controled by the government. It's suspected that the blackout was caused because she overruled technicians' opinion, looking for a solar % record. Not "enough thermal power stations" carefully avoids mentioning nuclear, that they want to erradicate. Private companies are requesting the records of CECOEL conversation to be published, some have leaked their logs. In one of those, REE technicians readily admit that oscilations are caused by not enough nuclear in the mix.

deleted 1y ago
[deleted]
toomuchtodo 1y ago
Doesn't have to be nuclear, can be anything that provides reactive power and grid services, i.e. grid forming battery storage. The problem is this service is needed on the Spanish grid, but isn't paid for in the market. Pay for it and the grid services will be provided. Resources below. Mind you, doesn't have to be a Tesla battery, just has to be a battery with power controls (grid forming) and some lithium cells for fast response (vs sodium for longer duration discharge). China launches world’s first grid-forming sodium-ion battery storage plant - https://www.ess-news.com/2025/06/03/china-launches-worlds-first-grid-forming-sodium-ion-battery-storage-plant/ https://www.ess-news.com/2025/06/03/china-launches-worlds-fi... - June 3rd, 2025 ("With a total investment of over CNY 460 million [$63.8 million] and occupying 34k square metres, the Baochi plant is designed for an installed capacity of 200MW/400MWh. Based on a dual daily charge-discharge cycle, it can regulate up to 580 GWh annually — enough to power 270,000 households, with 98 per cent of its energy sourced from renewables. The facility supports more than 30 local wind and solar power stations, alleviating the impact of intermittent supply and facilitating the integration of high shares of renewables into the grid.") Tesla Megapack in Texas Supports the Grid and Keeps the Lights On [Gambit Energy Storage] - https://www.tesla.com/videos/gambit-megapack https://www.tesla.com/videos/gambit-megapack - September 30, 2021 Tesla’s giant battery in Australia [Hornsdale Power Reserve] reduced grid service cost by 90% - https://news.ycombinator.com/item?id=17051066 https://news.ycombinator.com/item?id=17051066 - May 2018 Tesla Powerpacks Balancing the Grid in Terhills, Belgium [video] - https://www.youtube.com/watch?v=EVHQFrGzThg https://www.youtube.com/watch?v=EVHQFrGzThg - May 14, 2018

ZeroGravitas 1y ago

I think a gas plant lying about it's availability was eventually discovered to be a root cause of the South Australia blackout a few years back.

Rygian 1y ago

For me the keywords are "capacidad de control de tensión insuficiente" (insufficient voltage control capacity) which was already a known issue for years. REE asked back in 2019/2020 to create a new real-time dispatch mechanism for reactive power, because Spain still relies on fixed dispatch based on emails, phone calls, even snail mail. There was a pilot project in 2022 which was successful (dispatching in less than 5 seconds) but proved to be very expensive, and therefore incompatible with the European Union's directives on creating market-ready regulation processes [1]. [1][ES] https://www.eldiario.es/economia/competencia-reconocio-julio-2022-problemas-continuos-control-tension-red-electrica-peninsular_1_12299347.html https://www.eldiario.es/economia/competencia-reconocio-julio...

UltraSane 1y ago
"Spain still relies on fixed dispatch based on emails, phone calls, even snail mail. " Yikes!

demarq 1y ago

Wasn’t it supposed to be Russia

b3orn 1y ago
No, it was supposedly induced atmospheric vibration.

Dr_Knows 1y ago

@narag excellent recap. "carefully avoids mentioning nuclear" -- the nuclear operator/s decided not to supply electricitiy on that day.

cryptonector 1y ago

> REE did not have enough thermal power stations switched on when surge occurred That's.. that's a miscalculation alright. But it's not likely that it was just a small math error. It must have been policy to live on the edge.

Rygian 1y ago
It's common policy to plan your resource consumption in the most economical way. Couple that to a "next day" dispatching system. Couple that to resources that failed to do their job (did not compensate enough reactive power, or even added extra reactive power instead of compensating). Couple that to one of the most challenging electrical grids in the world. When taking that context in consideration, "living in the edge" is not really a legitimate way to put it.

mpweiher 1y ago

"REE did not have enough thermal power stations switched on when surge occurred"