20 ms·

What Happened in the UK Blackouts?

399 points by mitchoneill1 7y ago

pedrocr 7y ago
Isn't this kind of fault a good argument for synchronizing the UK grid to the rest of Europe? They seem to already have an interconnect that's of the scale of demand needed here, but it's a DC link.
- radiowave 7y ago
  If they were synchronized, it would mean that in the case of any larger failure that would draw too much current over the undersea link, the link itself would have to trip out.
  - pedrocr 7y ago
    That depends on the link size. But I guess it may very well be better to have a big enough DC link and have it automatically modulate power all the way up to maximum and then keep it there, instead of tripping.
Zenst 7y ago
"Frequency drops below 48.8Hz which triggers the Low Frequency Demand Disconnection scheme (LFDD). Approximately 1GW (5% of total) of Great Britain electricity demand is turned off." Is interesting as it highlights the importance of managed failure. Things fail, but the ability to see those failures and control how that fails is important. Had such an action not been taken, the result would of been a far greater outage/impact and more so an outage that you had no idea what parts it would impact.
- mytailorisrich 7y ago
  The issue with this specific outage is that it was badly managed. The main gripe people and authorities have with what happened is that key infrastructures immediately lost power. When (only) 5% of demand is turned off this should not impact railways, airports, and hospitals. Beyond the initial loss of power other problems amplified the consequences: For example, trains could not just restart but required specialist staff to attend each of them to get them through a specific procedure, which turned a short power loss into hours of delays. This highlights the importance of looking at systems globally. The grid is just one part of the system.
  - Zenst 7y ago
    Most situations/responses to an issue in hindsight can always be managed better, being able to have a cookie-cutter solution that fits most cases is better than none. THough I'm sure such plans will be refined - leasons learned. Thing is, a solution today might not be best for tomorrow as the grid's usage is not static, new this and that are added so the impact changes. It is interesting that you mention the inability for trains to restart. That was educational and was due to some new trains and showed that the drivers did not have the skills to in effect - reboot the trains from a cold start. So less a grid fault and more a case of that it never happened before so the result was less prepared. Interestingly the previous generation of trains did not have this issue, which may explain why it got overlooked as something that needed attention until such an event like this. As for hospitals, they all have backup generators, though if that generator fails to kick in as was the case in one I read about, that can and will cause issues. Again, may be a case of never needing it before and the combination of that and regular checks/tests presented the problem. Remember - nothing is perfect, and many lessons (as with any outage in all walks of life) will be learned from this as with all previous outages. But equally have to remember that whilst you can have the same combination of events happen in say 5 years time and have a response that works for that combination learned from the last time. Things change, like that new housing estate attached to the grid, small nuances like that could change everything or nothing. You just don't know. Heck, could be newer issues, suddenly find that you have thousands of electric cars being charged at the time you cut them off now feeding back small amounts of electricity back into the grid as they never expected to loose power without a physical disconnect.
    - mytailorisrich 7y ago
      All the issues I mentioned are obvious, and so are the potential problems you describe. This is not a question of "it's easy to say in hindsight" as these are the key issues that must be covered by standard planning. This blackout made a lot of noise because of this (apart from the actual disruption, obviously).
      - tripzilch 7y ago
- StreamBright 7y ago
  It also highlights one of the most important characteristics of energy grids: frequency sensitivity.
fyfy18 7y ago
What affect does the frequency drop have to home users? Is there protection in the substations near to homes that will cut off the power completely if the frequency goes out of range?
- bonzini 7y ago
  The main problem with frequency mismatch is that when two power sources "meet" with different frequency the effect can be catastrophic, because the faster generator pulls the slower (think of the "hares" in 3000-10000 m running competitions) and typically both end up breaking.
  - Zenst 7y ago
    Nice article covering frfequncy importance at grid level:- https://www.drax.com/energy-policy/need-whole-country-frequency/ https://www.drax.com/energy-policy/need-whole-country-freque... Though I'm not sure on the details of a frequency drop and impact upon your house in exacting details. Equally I'm not up on substations (did work on IT for large util company many decades ago, but not field engineering). But the first safety control from supply to the home, would be the distribution transformer and those back then had overheat protection. Also had a flag attached by wax at one spot, so if they did overheat and cut out for a while, you could visualy see that due to the flag being visible. But that was more a legacy solution that still prevailed into the 80's, unsure about now. But would not be supprised if such tried and tested things still take place as some form of backup monitoring. I learned about it after coding for a database that stored such records that had the field name SITESHIT, which was short for "SITES HIT", yes - you can see how that gave a young humble COBOL programmer a laugh. But it would be a count of how many times that wax flag had been triggered.
    - bonzini 7y ago
      That website has had me nerd-sniped. Another great one is on "black start": https://www.drax.com/technology/black-start-important-back-plan-youve-never-heard/ https://www.drax.com/technology/black-start-important-back-p...
kachnuv_ocasek 7y ago
What an interesting read! I had no idea that a blackout happened in UK recently, so I clicked out of curiosity and was rewarded with this delightful description of the initial events. It's also interesting to see the interplay of automated and manual systems in such a complicated scenario.
deleted 7y ago
[deleted]
eDISCO 7y ago
Most traditional generators (think CCGT, nuclear, hydro) run in synchronous mode. Which means the generators spins with rpm proportional/equal to the network frequency. For 50Hz that would be 3000rpm. When the total electricity demand exceeds the supply, the kinetic energy of spinning generators is converted to electricity which slows down the generator rpm, which reduces the frequency. All synchronous generators on a given network spin at the same frequency, so when the demand is too high all of them slow down a bit.
- circular_logic 7y ago
  That makes me wonder, if you were standing next to these turbines would you be able to here them slow down?
  - fanf2 7y ago
    The change in pitch from 50Hz to 48.8Hz was 0.4 semitones. 12 * ln 50 / ln 2 = 67.7 12 * ln 48.8 / ln 2 = 67.3
  - Iwan-Zotow 7y ago
    Yes
  - mnx 7y ago
    If I listen carefully, I can tell the difference between 48 and 50hz using a sine generator (although it's very subtle), so I'd guess that you possibly could.
  - bsder 7y ago
    Probably, although you are more likely to hear the 3000Hz to 2880Hz change in the generator RPM than the 50 to 48.8 change in the output.
    - deleted 7y ago
      [deleted]
  - kalleboo 7y ago
londons_explore 7y ago
It's notable that this report is completely consistent so far with the way I would design a cyber attack to take out a countries power grid. > The causes provided by RWE for the initiation of the trip of Little Barford steam turbine (ST1C) was due to discrepancies on three independent safety critical speed measurement signals to the generator control system. A cyber attack would most likely, rather than directly take capacity offline, instead make the system less stable by attempting to make safety systems not work properly. That has the benefit that the attack will be triggered at th exact same time as another incident, increasing the impact. Injecting false values for a sensor reading is a very good way to hide your tracks as an attacker - just inject a bunch of "0" values, and then uninstall whatever malware did it, and probably nobody will ever find out why that sensor randomly acted up.
- std_throwaway 7y ago
  An even more stealth method is to make the system work perfectly if everything is running fine. Like correctly reporting net frequency in the range 49-51 Hz but reporting false values when outside of that range. That way there will be no detectable errors until everything starts behaving erratically. The best way for a society to prepare is to have a few small incidents that put the safety systems to the test on a regular basis.
  - KyleJ61782 7y ago
    This. Very much this. In the presentations I've seen on the matter, the most vulnerable systems are those measuring and carrying telemetry back from various meters around the power system. One doesn't have to directly compromise the system to have an effect. Simply providing bad data to automated protection systems could be enough to cause instability.
    - aidos 7y ago
      A lesson Knight Capital learnt the hard way. https://en.m.wikipedia.org/wiki/Knight_Capital_Group https://en.m.wikipedia.org/wiki/Knight_Capital_Group
pm215 7y ago
The article says near the bottom "At the time [the GT1A generator tripped] there was 4000MW of unused generation in reserve, and if only a fraction of this reserve was instructed to dispatch within 60 seconds of the initial loss it’s very likely that load shedding would have not occurred." My understanding was that having reserves available within 30-60s was pretty expensive, and that that would have been what was used to handle the initial frequency drop (the interim report PDF says there was 1000MW of this). Drax's page on this (https://www.drax.com/technology/power-systems-super-subs/ https://www.drax.com/technology/power-systems-super-subs/) says that even spinning reserve can be on the order of two minutes, and the next tier below that (STOR) is on the order of 20 minutes. So does anybody know what the 4000MW available within 60s but not called on was?
- mitchoneill1 7y ago
  Hey! (author of the article here). I'm not happy with how I phrased this with the 4GW of capacity -- and while I suspect generation could have increased by hundreds of MW within 60 seconds I may be wrong. I've updated how I've phrased it in the blog to hopefully give a bit more clarity to my position. I don't meant to suggest that plants could ramp full within 60 seconds, it's that 4GW means there's a bunch of headroom, and that there was at least the physical possibility that dispatch instructions to increase generation/start-up (the reserves were "available") could have been sent out and plants could have started moving on an upward trajectory, creating even a few extra hundred MW of generation if many of the more flexible plants (gas, hydro) could increase their output even just a bit. Here in Australia we dispatch on a 5 minute period, and there are some plants that are very slow moving and have to ramp over tens of minutes or hours, but moving grid-wide generation up and down many hundreds of MW within a 5 minute dispatch period is a common occurrence, and so this coupled with the relatively high amount of reserve makes me suspect it would have at least been physically possible to increase output and hasten the return to a normal operating frequency. How you create both the systems and market structures/rules to oversee dispatch and coordination of something like that in a fast and automated way is very complex and infeasible to effectively do any time soon, but I think it's a system we should be thinking about and moving towards. Overall though I'm just grumpy about the amount of manual, slow work that's still involved in grid operations (when we lose an inter-connector here in Australia it takes 3-15 minutes for the dispatch engine to become aware that there's no inter-connector anymore as it needs to be reconfigured manually).
kzrdude 7y ago
At one stage, they have to shed load (turn power off), and that sounds totally reasonable. How are these areas divided, how does one turn power off for a whole area, and how is it decided who to cut from power (temporarily)?
- pbhjpbhj 7y ago
  From the little I know UK settlements have substations that serve something like 1000 homes. Older ones are open air affairs, newer ones are prefabbed enclosed buildings. They're oil cooled transformers and switching gear. One serves the village where my parents live, in a city there are two within about 400m of me (serving different neighbourhoods). Sometimes a neighbourhood will trip out, like in a thunderstorm; or get turned off (maintenance). I assume that the central grid has a switching order for these to be turned off to maintain essential supply. Factories, hospitals, and such have their own substations.
- pm215 7y ago
  https://www.westernpower.co.uk/downloads/4093 https://www.westernpower.co.uk/downloads/4093 is a pdf for one of the local power distribution companies that includes an explanation. The short answer is that they have frequency-sensitive relays that automatically disconnect some or all of the feeders downstream of 132kV substations as frequency drops through various specified limits.
hyperman1 7y ago
I like this article as it gives a lot of advanced insights about what seems at first an easy problem. Generators are tech from, what, 2 centuries ago? Keeping them in sync sounds boring, just talk together. And then, under the surface, looms an incredibly complex problem about working at scale, having engineering, political, organizational aspects.
jonatron 7y ago
It was a few minutes of downtime affecting a fraction of people. The last large UK outage was 10 years ago. The uptime has a lot of 9s already, so even if they ensure this particular problem wouldn't cause downtime again, the next time it'd just be a different rare event that causes it.
rwmj 7y ago
Why does frequency drop when the system is overloaded?
- OJFord 7y ago
  ELY5: When you're at a playground, and you're pushing the roundabout round and round, it gets harder and you get slower the more kids jump on it, right? ELY not 5, but I'm no expert either: in terms of phase sources like generators lead [0] sinks, by an amount proportional to their over-production; so the more sunk, the less they lead, and the lower the frequency. [0]: https://en.wikipedia.org/wiki/Leading_and_lagging_current https://en.wikipedia.org/wiki/Leading_and_lagging_current See also: https://en.wikipedia.org/wiki/Power_factor https://en.wikipedia.org/wiki/Power_factor
- techsupporter 7y ago
  Imagine if you are running on a conveyor belt at 60 steps per second. This belt is turning a crank that, let's say, moves water. You are the energy input (the sun for solar, heat energy for steam turbines, movement of wind for windmills, and so on), the crank is the generator, and the water is the load. Now imagine the water is suddenly turned into vegetable oil (more load has come on). It is more difficult to move. For a brief moment, your energy input is still the same but you can't push as much of the new liquid so your pace slows down a bit, to 58 steps per minute. You increase your exertion (add more energy input) and your pace climbs back to 60 steps per minute. Finally, imagine that you're running as hard as you can. That 60 steps per second to push that vegetable oil is as much as you can do. But the vegetable oil turns to heavy cream (even more load). You've put in as much energy as you can but your pace falters and drops to 49 steps per second. That's why the frequency falls.
- FPGAhacker 7y ago
  For the same reason a car engine’s RPMs will drop when you start going up a hill (unless you increase the throttle). Spinning up the generators, and keeping them spinning, takes energy on top of whatever the load is. If the load increases, but the input energy does not, the difference is robbed from the momentum of the spinning mass. This slows the generators down thus the frequency drops.
aurbano 7y ago
It actually sounds like this was really useful though, as a lot of the systems involved will be improved in the future. I'm not sure if the grid is already doing this, but they should trigger failures randomly, including Low Frequency Demand Disconnection schemes from time to time to ensure that all downstream systems are also ready (like the hospital someone mentioned that had an issue with the backup generator, or the trains that stopped working when the frequency failed). A bit like caos monkeys in dev environments.
- pm215 7y ago
  Triggering demand disconnection randomly is saying "we should provide consumers a service including deliberate random blackouts". That's OK in contexts like Netflix where the downstream systems are supposed to be able to be resilient to faults and failures are comparatively low-stakes anyway. But almost all grid consumers aren't resilient to power loss (would your house continue to be fully functional without noticeable ill effects if power to it went out? Mine wouldn't...) and even for industrial scale consumers who do have generators the costs of finding out in a deliberate blackout that their generator was faulty could be anything from expensive to life-threatening.
  - noir_lord 7y ago
    One of the hospitals affected by the blackout describe in this article had it's generators fail to come online cleanly, they had to go manual in ICU.
    - walshemj 7y ago
      Why are they not on battery at all times in ICU
  - phire 7y ago
    The question is not: Should we cut power to 5% of customers or not. The question is: Should we cat power to 5% of the customers now, or let the whole electrical grid stall, resulting in zero power for anyone.
- 7y ago
diggan 7y ago
To understand how/why these cascading failures can happen, Grady Hillhouse (Practical Engineering on YouTube) recently made a video about substations and what kind of faults can affect them, interesting if you're not super knowledgeable about our electric grids already: https://www.youtube.com/watch?v=7Q-aVBv7PWM https://www.youtube.com/watch?v=7Q-aVBv7PWM
- 4rt 7y ago
  I always upvote grady. It's great to see his subscriber count finally starting to match the quality of his videos.
abedci 7y ago
9k
stygiansonic 7y ago
For a related read: the Northeast blackout of 2003: https://en.m.wikipedia.org/wiki/Northeast_blackout_of_2003 https://en.m.wikipedia.org/wiki/Northeast_blackout_of_2003
techrich 7y ago
someone figured out that windmills dont turn all the time, so turned a coal power plant back on.
Havoc 7y ago
This to me suggests we need a hell of a lot more batteries in the system. Don't see anything else reacting fast enough.
computator 7y ago
Simplifying, this is what I get: (1) A lightning strike creates a transient voltage on the grid. (2) Some power generators remove themselves from the grid to prevent damage to themselves from the transient voltage. (3) Now there is too much demand on the grid -- too many people using power and not enough generators to supply it. (The line frequency dropping indicates that supply and demand are not balanced.) (4) So the grid operator begins to (deliberately) remove some people from the grid to keep the supply and demand balanced. What's not clear to me is what would happen if you didn't do step #4.
- NobodyNada 7y ago
  That’s when you end up with a cascading failure (https://en.wikipedia.org/wiki/Cascading_failure https://en.wikipedia.org/wiki/Cascading_failure). The remaining generators are overloaded by the demand and shut down to protect themselves as well, which further decreases the supply and causes more stations to shut off, etc. The Wikipedia article about the northeast US blackout of 2003 is a very interesting read: https://en.wikipedia.org/wiki/Northeast_blackout_of_2003 https://en.wikipedia.org/wiki/Northeast_blackout_of_2003
- philjohn 7y ago
  4 needs to happen, and is the benefit the UK has of a massive country-wide electricity grid. As someone else said, cascading failures and a much worse time.
effnorwood 7y ago
The power went out causing blackouts.
wyiske 7y ago
CityAM published a really interesting article about this https://www.cityam.com/what-was-behind-fridays-national-grid-outage-network-theory-not-conspiracy/amp/ https://www.cityam.com/what-was-behind-fridays-national-grid... It certainly fits the given explanation
hello_tyler 7y ago
That's just the government testing super sekret military tech. That or complete ineptitude and incompetence. I'm betting on the former. Nothing to see here. Did you know Donald Trump is the president of America???

399 points by mitchoneill1 7y ago

Isn't this kind of fault a good argument for synchronizing the UK grid to the rest of Europe? They seem to already have an interconnect that's of the scale of demand needed here, but it's a DC link.

radiowave 7y ago
If they were synchronized, it would mean that in the case of any larger failure that would draw too much current over the undersea link, the link itself would have to trip out.
- pedrocr 7y ago
  That depends on the link size. But I guess it may very well be better to have a big enough DC link and have it automatically modulate power all the way up to maximum and then keep it there, instead of tripping.

Zenst 7y ago

"Frequency drops below 48.8Hz which triggers the Low Frequency Demand Disconnection scheme (LFDD). Approximately 1GW (5% of total) of Great Britain electricity demand is turned off." Is interesting as it highlights the importance of managed failure. Things fail, but the ability to see those failures and control how that fails is important. Had such an action not been taken, the result would of been a far greater outage/impact and more so an outage that you had no idea what parts it would impact.

mytailorisrich 7y ago
The issue with this specific outage is that it was badly managed. The main gripe people and authorities have with what happened is that key infrastructures immediately lost power. When (only) 5% of demand is turned off this should not impact railways, airports, and hospitals. Beyond the initial loss of power other problems amplified the consequences: For example, trains could not just restart but required specialist staff to attend each of them to get them through a specific procedure, which turned a short power loss into hours of delays. This highlights the importance of looking at systems globally. The grid is just one part of the system.
- Zenst 7y ago
  Most situations/responses to an issue in hindsight can always be managed better, being able to have a cookie-cutter solution that fits most cases is better than none. THough I'm sure such plans will be refined - leasons learned. Thing is, a solution today might not be best for tomorrow as the grid's usage is not static, new this and that are added so the impact changes. It is interesting that you mention the inability for trains to restart. That was educational and was due to some new trains and showed that the drivers did not have the skills to in effect - reboot the trains from a cold start. So less a grid fault and more a case of that it never happened before so the result was less prepared. Interestingly the previous generation of trains did not have this issue, which may explain why it got overlooked as something that needed attention until such an event like this. As for hospitals, they all have backup generators, though if that generator fails to kick in as was the case in one I read about, that can and will cause issues. Again, may be a case of never needing it before and the combination of that and regular checks/tests presented the problem. Remember - nothing is perfect, and many lessons (as with any outage in all walks of life) will be learned from this as with all previous outages. But equally have to remember that whilst you can have the same combination of events happen in say 5 years time and have a response that works for that combination learned from the last time. Things change, like that new housing estate attached to the grid, small nuances like that could change everything or nothing. You just don't know. Heck, could be newer issues, suddenly find that you have thousands of electric cars being charged at the time you cut them off now feeding back small amounts of electricity back into the grid as they never expected to loose power without a physical disconnect.
  - mytailorisrich 7y ago
    All the issues I mentioned are obvious, and so are the potential problems you describe. This is not a question of "it's easy to say in hindsight" as these are the key issues that must be covered by standard planning. This blackout made a lot of noise because of this (apart from the actual disruption, obviously).
    - tripzilch 7y ago
StreamBright 7y ago
It also highlights one of the most important characteristics of energy grids: frequency sensitivity.

fyfy18 7y ago

What affect does the frequency drop have to home users? Is there protection in the substations near to homes that will cut off the power completely if the frequency goes out of range?

bonzini 7y ago
The main problem with frequency mismatch is that when two power sources "meet" with different frequency the effect can be catastrophic, because the faster generator pulls the slower (think of the "hares" in 3000-10000 m running competitions) and typically both end up breaking.
- Zenst 7y ago
  Nice article covering frfequncy importance at grid level:- https://www.drax.com/energy-policy/need-whole-country-frequency/ https://www.drax.com/energy-policy/need-whole-country-freque... Though I'm not sure on the details of a frequency drop and impact upon your house in exacting details. Equally I'm not up on substations (did work on IT for large util company many decades ago, but not field engineering). But the first safety control from supply to the home, would be the distribution transformer and those back then had overheat protection. Also had a flag attached by wax at one spot, so if they did overheat and cut out for a while, you could visualy see that due to the flag being visible. But that was more a legacy solution that still prevailed into the 80's, unsure about now. But would not be supprised if such tried and tested things still take place as some form of backup monitoring. I learned about it after coding for a database that stored such records that had the field name SITESHIT, which was short for "SITES HIT", yes - you can see how that gave a young humble COBOL programmer a laugh. But it would be a count of how many times that wax flag had been triggered.
  - bonzini 7y ago
    That website has had me nerd-sniped. Another great one is on "black start": https://www.drax.com/technology/black-start-important-back-plan-youve-never-heard/ https://www.drax.com/technology/black-start-important-back-p...

kachnuv_ocasek 7y ago

What an interesting read! I had no idea that a blackout happened in UK recently, so I clicked out of curiosity and was rewarded with this delightful description of the initial events. It's also interesting to see the interplay of automated and manual systems in such a complicated scenario.

deleted 7y ago

[deleted]

eDISCO 7y ago

Most traditional generators (think CCGT, nuclear, hydro) run in synchronous mode. Which means the generators spins with rpm proportional/equal to the network frequency. For 50Hz that would be 3000rpm. When the total electricity demand exceeds the supply, the kinetic energy of spinning generators is converted to electricity which slows down the generator rpm, which reduces the frequency. All synchronous generators on a given network spin at the same frequency, so when the demand is too high all of them slow down a bit.

circular_logic 7y ago
That makes me wonder, if you were standing next to these turbines would you be able to here them slow down?
- fanf2 7y ago
  The change in pitch from 50Hz to 48.8Hz was 0.4 semitones. 12 * ln 50 / ln 2 = 67.7 12 * ln 48.8 / ln 2 = 67.3
- Iwan-Zotow 7y ago
  Yes
- mnx 7y ago
  If I listen carefully, I can tell the difference between 48 and 50hz using a sine generator (although it's very subtle), so I'd guess that you possibly could.
- bsder 7y ago
  Probably, although you are more likely to hear the 3000Hz to 2880Hz change in the generator RPM than the 50 to 48.8 change in the output.
  - deleted 7y ago
    [deleted]
- kalleboo 7y ago

londons_explore 7y ago

It's notable that this report is completely consistent so far with the way I would design a cyber attack to take out a countries power grid. > The causes provided by RWE for the initiation of the trip of Little Barford steam turbine (ST1C) was due to discrepancies on three independent safety critical speed measurement signals to the generator control system. A cyber attack would most likely, rather than directly take capacity offline, instead make the system less stable by attempting to make safety systems not work properly. That has the benefit that the attack will be triggered at th exact same time as another incident, increasing the impact. Injecting false values for a sensor reading is a very good way to hide your tracks as an attacker - just inject a bunch of "0" values, and then uninstall whatever malware did it, and probably nobody will ever find out why that sensor randomly acted up.

std_throwaway 7y ago
An even more stealth method is to make the system work perfectly if everything is running fine. Like correctly reporting net frequency in the range 49-51 Hz but reporting false values when outside of that range. That way there will be no detectable errors until everything starts behaving erratically. The best way for a society to prepare is to have a few small incidents that put the safety systems to the test on a regular basis.
- KyleJ61782 7y ago
  This. Very much this. In the presentations I've seen on the matter, the most vulnerable systems are those measuring and carrying telemetry back from various meters around the power system. One doesn't have to directly compromise the system to have an effect. Simply providing bad data to automated protection systems could be enough to cause instability.
  - aidos 7y ago
    A lesson Knight Capital learnt the hard way. https://en.m.wikipedia.org/wiki/Knight_Capital_Group https://en.m.wikipedia.org/wiki/Knight_Capital_Group

pm215 7y ago

The article says near the bottom "At the time [the GT1A generator tripped] there was 4000MW of unused generation in reserve, and if only a fraction of this reserve was instructed to dispatch within 60 seconds of the initial loss it’s very likely that load shedding would have not occurred." My understanding was that having reserves available within 30-60s was pretty expensive, and that that would have been what was used to handle the initial frequency drop (the interim report PDF says there was 1000MW of this). Drax's page on this (https://www.drax.com/technology/power-systems-super-subs/ https://www.drax.com/technology/power-systems-super-subs/) says that even spinning reserve can be on the order of two minutes, and the next tier below that (STOR) is on the order of 20 minutes. So does anybody know what the 4000MW available within 60s but not called on was?

mitchoneill1 7y ago
Hey! (author of the article here). I'm not happy with how I phrased this with the 4GW of capacity -- and while I suspect generation could have increased by hundreds of MW within 60 seconds I may be wrong. I've updated how I've phrased it in the blog to hopefully give a bit more clarity to my position. I don't meant to suggest that plants could ramp full within 60 seconds, it's that 4GW means there's a bunch of headroom, and that there was at least the physical possibility that dispatch instructions to increase generation/start-up (the reserves were "available") could have been sent out and plants could have started moving on an upward trajectory, creating even a few extra hundred MW of generation if many of the more flexible plants (gas, hydro) could increase their output even just a bit. Here in Australia we dispatch on a 5 minute period, and there are some plants that are very slow moving and have to ramp over tens of minutes or hours, but moving grid-wide generation up and down many hundreds of MW within a 5 minute dispatch period is a common occurrence, and so this coupled with the relatively high amount of reserve makes me suspect it would have at least been physically possible to increase output and hasten the return to a normal operating frequency. How you create both the systems and market structures/rules to oversee dispatch and coordination of something like that in a fast and automated way is very complex and infeasible to effectively do any time soon, but I think it's a system we should be thinking about and moving towards. Overall though I'm just grumpy about the amount of manual, slow work that's still involved in grid operations (when we lose an inter-connector here in Australia it takes 3-15 minutes for the dispatch engine to become aware that there's no inter-connector anymore as it needs to be reconfigured manually).

kzrdude 7y ago

At one stage, they have to shed load (turn power off), and that sounds totally reasonable. How are these areas divided, how does one turn power off for a whole area, and how is it decided who to cut from power (temporarily)?

pbhjpbhj 7y ago
From the little I know UK settlements have substations that serve something like 1000 homes. Older ones are open air affairs, newer ones are prefabbed enclosed buildings. They're oil cooled transformers and switching gear. One serves the village where my parents live, in a city there are two within about 400m of me (serving different neighbourhoods). Sometimes a neighbourhood will trip out, like in a thunderstorm; or get turned off (maintenance). I assume that the central grid has a switching order for these to be turned off to maintain essential supply. Factories, hospitals, and such have their own substations.
pm215 7y ago
https://www.westernpower.co.uk/downloads/4093 https://www.westernpower.co.uk/downloads/4093 is a pdf for one of the local power distribution companies that includes an explanation. The short answer is that they have frequency-sensitive relays that automatically disconnect some or all of the feeders downstream of 132kV substations as frequency drops through various specified limits.

hyperman1 7y ago

I like this article as it gives a lot of advanced insights about what seems at first an easy problem. Generators are tech from, what, 2 centuries ago? Keeping them in sync sounds boring, just talk together. And then, under the surface, looms an incredibly complex problem about working at scale, having engineering, political, organizational aspects.

jonatron 7y ago

It was a few minutes of downtime affecting a fraction of people. The last large UK outage was 10 years ago. The uptime has a lot of 9s already, so even if they ensure this particular problem wouldn't cause downtime again, the next time it'd just be a different rare event that causes it.

rwmj 7y ago

Why does frequency drop when the system is overloaded?

OJFord 7y ago
ELY5: When you're at a playground, and you're pushing the roundabout round and round, it gets harder and you get slower the more kids jump on it, right? ELY not 5, but I'm no expert either: in terms of phase sources like generators lead [0] sinks, by an amount proportional to their over-production; so the more sunk, the less they lead, and the lower the frequency. [0]: https://en.wikipedia.org/wiki/Leading_and_lagging_current https://en.wikipedia.org/wiki/Leading_and_lagging_current See also: https://en.wikipedia.org/wiki/Power_factor https://en.wikipedia.org/wiki/Power_factor
techsupporter 7y ago
Imagine if you are running on a conveyor belt at 60 steps per second. This belt is turning a crank that, let's say, moves water. You are the energy input (the sun for solar, heat energy for steam turbines, movement of wind for windmills, and so on), the crank is the generator, and the water is the load. Now imagine the water is suddenly turned into vegetable oil (more load has come on). It is more difficult to move. For a brief moment, your energy input is still the same but you can't push as much of the new liquid so your pace slows down a bit, to 58 steps per minute. You increase your exertion (add more energy input) and your pace climbs back to 60 steps per minute. Finally, imagine that you're running as hard as you can. That 60 steps per second to push that vegetable oil is as much as you can do. But the vegetable oil turns to heavy cream (even more load). You've put in as much energy as you can but your pace falters and drops to 49 steps per second. That's why the frequency falls.
FPGAhacker 7y ago
For the same reason a car engine’s RPMs will drop when you start going up a hill (unless you increase the throttle). Spinning up the generators, and keeping them spinning, takes energy on top of whatever the load is. If the load increases, but the input energy does not, the difference is robbed from the momentum of the spinning mass. This slows the generators down thus the frequency drops.

aurbano 7y ago

It actually sounds like this was really useful though, as a lot of the systems involved will be improved in the future. I'm not sure if the grid is already doing this, but they should trigger failures randomly, including Low Frequency Demand Disconnection schemes from time to time to ensure that all downstream systems are also ready (like the hospital someone mentioned that had an issue with the backup generator, or the trains that stopped working when the frequency failed). A bit like caos monkeys in dev environments.

pm215 7y ago
Triggering demand disconnection randomly is saying "we should provide consumers a service including deliberate random blackouts". That's OK in contexts like Netflix where the downstream systems are supposed to be able to be resilient to faults and failures are comparatively low-stakes anyway. But almost all grid consumers aren't resilient to power loss (would your house continue to be fully functional without noticeable ill effects if power to it went out? Mine wouldn't...) and even for industrial scale consumers who do have generators the costs of finding out in a deliberate blackout that their generator was faulty could be anything from expensive to life-threatening.
- noir_lord 7y ago
  One of the hospitals affected by the blackout describe in this article had it's generators fail to come online cleanly, they had to go manual in ICU.
  - walshemj 7y ago
    Why are they not on battery at all times in ICU
- phire 7y ago
  The question is not: Should we cut power to 5% of customers or not. The question is: Should we cat power to 5% of the customers now, or let the whole electrical grid stall, resulting in zero power for anyone.
7y ago

diggan 7y ago

To understand how/why these cascading failures can happen, Grady Hillhouse (Practical Engineering on YouTube) recently made a video about substations and what kind of faults can affect them, interesting if you're not super knowledgeable about our electric grids already: https://www.youtube.com/watch?v=7Q-aVBv7PWM https://www.youtube.com/watch?v=7Q-aVBv7PWM

4rt 7y ago
I always upvote grady. It's great to see his subscriber count finally starting to match the quality of his videos.

abedci 7y ago

stygiansonic 7y ago

For a related read: the Northeast blackout of 2003: https://en.m.wikipedia.org/wiki/Northeast_blackout_of_2003 https://en.m.wikipedia.org/wiki/Northeast_blackout_of_2003

techrich 7y ago

someone figured out that windmills dont turn all the time, so turned a coal power plant back on.

Havoc 7y ago

This to me suggests we need a hell of a lot more batteries in the system. Don't see anything else reacting fast enough.

computator 7y ago

Simplifying, this is what I get: (1) A lightning strike creates a transient voltage on the grid. (2) Some power generators remove themselves from the grid to prevent damage to themselves from the transient voltage. (3) Now there is too much demand on the grid -- too many people using power and not enough generators to supply it. (The line frequency dropping indicates that supply and demand are not balanced.) (4) So the grid operator begins to (deliberately) remove some people from the grid to keep the supply and demand balanced. What's not clear to me is what would happen if you didn't do step #4.

NobodyNada 7y ago
That’s when you end up with a cascading failure (https://en.wikipedia.org/wiki/Cascading_failure https://en.wikipedia.org/wiki/Cascading_failure). The remaining generators are overloaded by the demand and shut down to protect themselves as well, which further decreases the supply and causes more stations to shut off, etc. The Wikipedia article about the northeast US blackout of 2003 is a very interesting read: https://en.wikipedia.org/wiki/Northeast_blackout_of_2003 https://en.wikipedia.org/wiki/Northeast_blackout_of_2003
philjohn 7y ago
4 needs to happen, and is the benefit the UK has of a massive country-wide electricity grid. As someone else said, cascading failures and a much worse time.

effnorwood 7y ago

The power went out causing blackouts.

wyiske 7y ago

CityAM published a really interesting article about this https://www.cityam.com/what-was-behind-fridays-national-grid-outage-network-theory-not-conspiracy/amp/ https://www.cityam.com/what-was-behind-fridays-national-grid... It certainly fits the given explanation

hello_tyler 7y ago

That's just the government testing super sekret military tech. That or complete ineptitude and incompetence. I'm betting on the former. Nothing to see here. Did you know Donald Trump is the president of America???