5 ms·

Seems the money shot starts at page 131: > The ultimate cause of the peninsular electrical zero on April 28th was a phenomenon of overvoltages in the form of a

by diggan 1y ago

Seems the money shot starts at page 131: > The ultimate cause of the peninsular electrical zero on April 28th was a phenomenon of overvoltages in the form of a "chain reaction" in which high voltages cause generation disconnections, which in turn causes new increases in voltage and thus new disconnections, and so on. > 1. The system showed insufficient dynamic voltage control capabilities sufficient to maintain stable voltage > 2. A series of rhythmic oscillations significantly conditioned the system, modifying its configuration and increasing the difficulties for voltage stabilization. If I understand it correctly (and like software, typical), it was a positive feedback-loop. Since there wasn't enough voltage control, some other station had to be added but got overloaded instead, also turning off, and then on to the next station. Late addition: It was very helpful for me to read through the "ANNEX X. BRIEF BASICS OF THE ELECTRIC SYSTEM" (page 168) before trying to read the report itself, as it explains a lot of things that the rest of the report (rightly) assumes you already know.

leymed 1y ago
I think your interpretation is correct. The voltage control is done at the high level of the grid, meaning the control covers bigger generation stations and major substations. Even if it’s small generator, rotating machinery, you won’t have strict voltage control other than its own AVR. The problem I see here is that we embed smaller individual generations at the lower level, where they pump the generated power to the grid at the medium voltage level. When you have majority of your generation at this level, you won’t have strict control over voltage and even frequency, I assume. I’m still digesting the report, but what I am after is whether they really neglected it and if it is not possible to do voltage control with 50% generation coming from renewable and through medium voltage level, aka lower level.
amluto 1y ago
I’m a bit mystified as to how the grid controls voltage at all. The non-renewable plants follow the rules here: https://www.boe.es/buscar/doc.php?id=BOE-A-2000-5204 https://www.boe.es/buscar/doc.php?id=BOE-A-2000-5204 7.1(b) seems to be saying that generators connected at 200kV adjust their reactive power generation/absorption in real time according to the voltage they observe, based on a lookup table provided by the grid operator. This seems sort of sensible according to my limited understanding of the theory of AC grids. You can write some differential equations and pretend everything is continuous (as opposed to being a LUT with 11 steps or so), and you can determine that the grid is stable. However, check out this shorter report from red eléctrica: https://d1n1o4zeyfu21r.cloudfront.net/WEB_Incident_%2028A_SpanishPeninsularElectricalSystem_18june25.pdf https://d1n1o4zeyfu21r.cloudfront.net/WEB_Incident_%2028A_Sp... Apparently these 220kV plants are connected to the 400kV grid via transformers in substations that are not owned by the generator operators. And those transformers have “tap changers” that attempt to keep the 220kV secondary side at the correct voltage within some fairly large voltage range on the 400kV side. Won’t this defeat the voltage control that the 220kV generators are supposed to provide? If the grid voltage is high, then absorption of reactive power is needed [0], and the generators are supposed to determine that they need to absorb reactive power (which they can do), but if the tap changer changes its setting, then the generator will not react correctly to the voltage on the 400kV side. In other words, one would like the generator to absorb reactive power according to P_reactive(primary voltage • 220/400), but the actual behavior is P_reactive(primary voltage • 220/400 • tap changer position), the tap changer position is presumably something like 400/primary voltage, and I don’t understand how the result is supposed to function in any useful way. Adding insult to injury, the red eléctrica repoet authors seem to be suggesting that a bunch of tap changers operators didn’t configure their tap changes well enough to even keep secondary voltages in range. Does anyone with more familiarity with these systems know how they’re supposed to work? [0] I can never remember the sign convention for reactive power.
- jakewins 1y ago
  I don't claim to know the details of reactive power management, but the primary mechanisms for grid stability in the EU is the "cascade" of services the TSOs procures: - Fast Frequency Response (FFR), sub-second power adjustment following frequency table - Frequency Containment Reserve (FCR), ~second power adjustment following frequency table - Automatic Frequency Restoration Reserve (aFRR), ~second energy production following TSO setpoint signal - Manual Frequency Restoration Reserve, ~minute energy production following TSO activation signals My understanding is the primary failure in Spain was that 9 separate synchronous plants that had sold aFRR(?) to the TSO then failed to deliver, so when the TSO algorithms tried to adjust the oscillations, nothing happened. Everything else was kinda "as designed".
  - pjc50 1y ago
    > 9 separate synchronous plants that had sold aFRR(?) to the TSO then failed to deliver, so when the TSO algorithms tried to adjust the oscillations, nothing happened. Oof. This sounds like a classic of "it's only needed in emergencies, so it's only in emergencies that we find out it doesn't work".
    - jakewins 1y ago
      I don't know about the Spanish market, but at least in the markets I'm involved in aFRR is an "always on" product, the TSO controls your plant with a setpoint that updates in near-real-time throughout the period you've sold to them.. it's not clear to me that the product that wasn't delivered was actually aFRR though, maybe it was something else less frequently called upon.
  - amluto 1y ago
    It likes to me like a major factor was that the grid failed to control voltage, not frequency. Frequency control should be unaffected by transformers.
- scrlk 1y ago
  The automatic voltage regulator (AVR) of the generator and the on-load tap changer (OLTC) operate on different timescales (AVR = quick; OLTC = slow). OLTCs are typically set up with a voltage deadband and a time delay to prevent 'hunting' (i.e., repeatedly tapping up and down).
madaxe_again 1y ago
I’ve just read the whole shebang. While the overall reason for the mass failure you cite is correct - a cascading failure - the interesting bit here are the oscillations that lead to it. It looks very much like this was driven by algorithmic volatility trading of electricity spots - overproduction, price goes negative, buys placed, production ramps in response to rising price, price rises, sells placed, production falls due to falling price. The period of the oscillations in the grid seen before the blackout suggest a relatively slow cycle, and what they describe in the report sounds very much like this was an interaction between price-driven supply and real world supply. It does speak to there being inadequate storage available on the grid to smooth demand and therefore pricing, but it also suggests that in certain conditions a harmonic can be set up between the market and price-driven production with catastrophic consequences.
- diggan 1y ago
  > It looks very much like this was driven by algorithmic volatility trading of electricity spots Yes, + less "reactive power stations" than expected was available (seems the day some unexpectedly went offline, and not enough safeguards/communication to realize this) + a switch between the French import/export that happened at the same time, leading to the overvoltage issue, which then spiraled. As far as I read the report, there were multiple causes, not a single one like "algorithmic volatility trading of electricity spots" but a combination of the issues where one-by-one, things would have been fine but all together? Shit broke

by diggan 1y ago

leymed 1y ago

I think your interpretation is correct. The voltage control is done at the high level of the grid, meaning the control covers bigger generation stations and major substations. Even if it’s small generator, rotating machinery, you won’t have strict voltage control other than its own AVR. The problem I see here is that we embed smaller individual generations at the lower level, where they pump the generated power to the grid at the medium voltage level. When you have majority of your generation at this level, you won’t have strict control over voltage and even frequency, I assume. I’m still digesting the report, but what I am after is whether they really neglected it and if it is not possible to do voltage control with 50% generation coming from renewable and through medium voltage level, aka lower level.

amluto 1y ago

I’m a bit mystified as to how the grid controls voltage at all. The non-renewable plants follow the rules here: https://www.boe.es/buscar/doc.php?id=BOE-A-2000-5204 https://www.boe.es/buscar/doc.php?id=BOE-A-2000-5204 7.1(b) seems to be saying that generators connected at 200kV adjust their reactive power generation/absorption in real time according to the voltage they observe, based on a lookup table provided by the grid operator. This seems sort of sensible according to my limited understanding of the theory of AC grids. You can write some differential equations and pretend everything is continuous (as opposed to being a LUT with 11 steps or so), and you can determine that the grid is stable. However, check out this shorter report from red eléctrica: https://d1n1o4zeyfu21r.cloudfront.net/WEB_Incident_%2028A_SpanishPeninsularElectricalSystem_18june25.pdf https://d1n1o4zeyfu21r.cloudfront.net/WEB_Incident_%2028A_Sp... Apparently these 220kV plants are connected to the 400kV grid via transformers in substations that are not owned by the generator operators. And those transformers have “tap changers” that attempt to keep the 220kV secondary side at the correct voltage within some fairly large voltage range on the 400kV side. Won’t this defeat the voltage control that the 220kV generators are supposed to provide? If the grid voltage is high, then absorption of reactive power is needed [0], and the generators are supposed to determine that they need to absorb reactive power (which they can do), but if the tap changer changes its setting, then the generator will not react correctly to the voltage on the 400kV side. In other words, one would like the generator to absorb reactive power according to P_reactive(primary voltage • 220/400), but the actual behavior is P_reactive(primary voltage • 220/400 • tap changer position), the tap changer position is presumably something like 400/primary voltage, and I don’t understand how the result is supposed to function in any useful way. Adding insult to injury, the red eléctrica repoet authors seem to be suggesting that a bunch of tap changers operators didn’t configure their tap changes well enough to even keep secondary voltages in range. Does anyone with more familiarity with these systems know how they’re supposed to work? [0] I can never remember the sign convention for reactive power.

jakewins 1y ago
I don't claim to know the details of reactive power management, but the primary mechanisms for grid stability in the EU is the "cascade" of services the TSOs procures: - Fast Frequency Response (FFR), sub-second power adjustment following frequency table - Frequency Containment Reserve (FCR), ~second power adjustment following frequency table - Automatic Frequency Restoration Reserve (aFRR), ~second energy production following TSO setpoint signal - Manual Frequency Restoration Reserve, ~minute energy production following TSO activation signals My understanding is the primary failure in Spain was that 9 separate synchronous plants that had sold aFRR(?) to the TSO then failed to deliver, so when the TSO algorithms tried to adjust the oscillations, nothing happened. Everything else was kinda "as designed".
- pjc50 1y ago
  > 9 separate synchronous plants that had sold aFRR(?) to the TSO then failed to deliver, so when the TSO algorithms tried to adjust the oscillations, nothing happened. Oof. This sounds like a classic of "it's only needed in emergencies, so it's only in emergencies that we find out it doesn't work".
  - jakewins 1y ago
    I don't know about the Spanish market, but at least in the markets I'm involved in aFRR is an "always on" product, the TSO controls your plant with a setpoint that updates in near-real-time throughout the period you've sold to them.. it's not clear to me that the product that wasn't delivered was actually aFRR though, maybe it was something else less frequently called upon.
- amluto 1y ago
  It likes to me like a major factor was that the grid failed to control voltage, not frequency. Frequency control should be unaffected by transformers.
scrlk 1y ago
The automatic voltage regulator (AVR) of the generator and the on-load tap changer (OLTC) operate on different timescales (AVR = quick; OLTC = slow). OLTCs are typically set up with a voltage deadband and a time delay to prevent 'hunting' (i.e., repeatedly tapping up and down).

madaxe_again 1y ago

I’ve just read the whole shebang. While the overall reason for the mass failure you cite is correct - a cascading failure - the interesting bit here are the oscillations that lead to it. It looks very much like this was driven by algorithmic volatility trading of electricity spots - overproduction, price goes negative, buys placed, production ramps in response to rising price, price rises, sells placed, production falls due to falling price. The period of the oscillations in the grid seen before the blackout suggest a relatively slow cycle, and what they describe in the report sounds very much like this was an interaction between price-driven supply and real world supply. It does speak to there being inadequate storage available on the grid to smooth demand and therefore pricing, but it also suggests that in certain conditions a harmonic can be set up between the market and price-driven production with catastrophic consequences.

diggan 1y ago
> It looks very much like this was driven by algorithmic volatility trading of electricity spots Yes, + less "reactive power stations" than expected was available (seems the day some unexpectedly went offline, and not enough safeguards/communication to realize this) + a switch between the French import/export that happened at the same time, leading to the overvoltage issue, which then spiraled. As far as I read the report, there were multiple causes, not a single one like "algorithmic volatility trading of electricity spots" but a combination of the issues where one-by-one, things would have been fine but all together? Shit broke