4 ms·

It actually sounds like this was really useful though, as a lot of the systems involved will be improved in the future. I'm not sure if the grid is already doi

It actually sounds like this was really useful though, as a lot of the systems involved will be improved in the future. I'm not sure if the grid is already doing this, but they should trigger failures randomly, including Low Frequency Demand Disconnection schemes from time to time to ensure that all downstream systems are also ready (like the hospital someone mentioned that had an issue with the backup generator, or the trains that stopped working when the frequency failed). A bit like caos monkeys in dev environments.

pm215 7y ago
Triggering demand disconnection randomly is saying "we should provide consumers a service including deliberate random blackouts". That's OK in contexts like Netflix where the downstream systems are supposed to be able to be resilient to faults and failures are comparatively low-stakes anyway. But almost all grid consumers aren't resilient to power loss (would your house continue to be fully functional without noticeable ill effects if power to it went out? Mine wouldn't...) and even for industrial scale consumers who do have generators the costs of finding out in a deliberate blackout that their generator was faulty could be anything from expensive to life-threatening.
- noir_lord 7y ago
  One of the hospitals affected by the blackout describe in this article had it's generators fail to come online cleanly, they had to go manual in ICU.
  - walshemj 7y ago
    Why are they not on battery at all times in ICU
- phire 7y ago
  The question is not: Should we cut power to 5% of customers or not. The question is: Should we cat power to 5% of the customers now, or let the whole electrical grid stall, resulting in zero power for anyone.
lozenge 7y ago
Doesn't LFDD just mean cutting off large areas from the grid to improve the frequency elsewhere? I doubt it would be accepted by home users and businesses. Hospitals should be running their own local tests.
bartwe 7y ago
Well this was a chaosmonkey (the lightning) injecting a random fault ;)
buckminster 7y ago
The whole point of sophisticated grid management is to save us money by reducing the total generating capacity needed for a reliable supply. If this were to impose massive costs on the rest of society in the form of planned blackouts then what would be the point? Why not invest in a little extra capacity?
xyk2 7y ago
That would be like AWS telling all customers that their services would randomly go offline to test disaster response. The responsibility for blackout testing should be at the client level, at each hospital, train station etc similar to fire drills.
danpalmer 7y ago
I assumed that the LFDD scheme was a small number of large power users such as factories that would get cheaper rates, much like AWS spot instances. Another commenter has said this is how it works in Germany. Unfortunately, some quick research[1] suggests this isn’t the case, and actually it’s just a contractual obligation of the power companies to disconnect enough demand to bring power use back down. The amount they need to disconnect is banded by the frequency the grid is at, and apparently done at the substation level. Some industries/applications such as infrastructure, hospitals, airports, etc, is supposed to be set up to never disconnect in these scenarios, but it seems this failed. [1]: https://www.westernpower.co.uk/downloads/4093 https://www.westernpower.co.uk/downloads/4093
- pm215 7y ago
  The UK has the 'pay commercial users who are able to voluntarily reduce demand' system too, and that was part of the initial response to the frequency drop, I think. The problem is that the volume of demand you can get rid of this way isn't necessarily enough to be sufficient -- if you need to dump 10 or 20% of demand and especially if it needs to be spread evenly over the grid then it's going to be hard to do without disconnecting people involuntarily. I wouldn't be surprised if we could do better in the volume of demand we could reduce on demand, especially with smarter technology, though. At some point the question is whether the extra complexity and infrastructure is cost-effective to avoid once-a-decade events on the scale of this one, which probably 90% of the UK would never have noticed unless they read the news...
Slartie 7y ago
Sorry, but the electrical grid is not built like a Netflix server farm. Its explicit purpose is to provide reliability to the customer, so the customer will not have to invest into mitigations for network failures, because it is much cheaper, more economical and technically much more feasible to make these investments in the network than individually on every single customer location. The only exception are some very specific customers for which the provided amount of reliability is not enough - these need to increase the reliability by investing in additional failsafes, and are of course responsible for testing these by themselves (if they'd like to have random network outages performed, they are free to do so, because any consequences from these harsh tests are in their responsibility anyway, just as any consequences from not doing proper tests and thus having the safeguards potentially fail in an emergency). Netflix also would never intentionally break my FireTV box, which I use to watch Netflix, just to ensure that I have a backup in place (which I don't, because I don't care THAT much about Netflix availability). [edit] Now that I think about it...the purpose of the Netflix server farm is also to provide reliability to the customer, so it's actually comparable to a certain degree. It's just that the possible outcome of a mitigation failure in the Netflix server farm is limited to some people not watching Netflix for a few minutes, while the possible outcome of a power outage even for residential customers are usually far worse, let alone industrial or health customers. This allows Netflix to take the limited risk of intentionally breaking internals in their infrastructure, because the worst that can happen is them having to excuse themselves to some customers for a service interruption. Also, no one expects them to submit a report describing how it came to those interruptions, so they don't need to admit that they were the result of intentional self-inflicted disruptions (which sounds really crazy and irresponsible to pretty much all consumers out there). With electrical grid operators, the stakes are infinitely higher, and they are legally required to investigate any bigger problems and report details about their origins to both government authorities and customers. And you can be sure that "oh we just intentionally dropped 5% of the network to see if we get a full blackout or if our failsafes work" would not be an acceptable excuse to any of these two.
7952 7y ago
The best time to test a generator backup is when the grid power is still available. So that you can revert if the generator fails to start.
rossdavidh 7y ago
You would be setting up a Chaos Monkey (https://en.wikipedia.org/wiki/Chaos_engineering#Chaos_Monkey https://en.wikipedia.org/wiki/Chaos_engineering#Chaos_Monkey) on your electrical grid. The problem is, while these kinds of things are an awesomely good idea early in a system's life, they are very difficult to put in after it has become both complex and mission-critical.

pm215 7y ago

Triggering demand disconnection randomly is saying "we should provide consumers a service including deliberate random blackouts". That's OK in contexts like Netflix where the downstream systems are supposed to be able to be resilient to faults and failures are comparatively low-stakes anyway. But almost all grid consumers aren't resilient to power loss (would your house continue to be fully functional without noticeable ill effects if power to it went out? Mine wouldn't...) and even for industrial scale consumers who do have generators the costs of finding out in a deliberate blackout that their generator was faulty could be anything from expensive to life-threatening.

noir_lord 7y ago
One of the hospitals affected by the blackout describe in this article had it's generators fail to come online cleanly, they had to go manual in ICU.
- walshemj 7y ago
  Why are they not on battery at all times in ICU
phire 7y ago
The question is not: Should we cut power to 5% of customers or not. The question is: Should we cat power to 5% of the customers now, or let the whole electrical grid stall, resulting in zero power for anyone.

lozenge 7y ago

Doesn't LFDD just mean cutting off large areas from the grid to improve the frequency elsewhere? I doubt it would be accepted by home users and businesses. Hospitals should be running their own local tests.

bartwe 7y ago

Well this was a chaosmonkey (the lightning) injecting a random fault ;)

buckminster 7y ago

The whole point of sophisticated grid management is to save us money by reducing the total generating capacity needed for a reliable supply. If this were to impose massive costs on the rest of society in the form of planned blackouts then what would be the point? Why not invest in a little extra capacity?

xyk2 7y ago

That would be like AWS telling all customers that their services would randomly go offline to test disaster response. The responsibility for blackout testing should be at the client level, at each hospital, train station etc similar to fire drills.

danpalmer 7y ago

I assumed that the LFDD scheme was a small number of large power users such as factories that would get cheaper rates, much like AWS spot instances. Another commenter has said this is how it works in Germany. Unfortunately, some quick research[1] suggests this isn’t the case, and actually it’s just a contractual obligation of the power companies to disconnect enough demand to bring power use back down. The amount they need to disconnect is banded by the frequency the grid is at, and apparently done at the substation level. Some industries/applications such as infrastructure, hospitals, airports, etc, is supposed to be set up to never disconnect in these scenarios, but it seems this failed. [1]: https://www.westernpower.co.uk/downloads/4093 https://www.westernpower.co.uk/downloads/4093

pm215 7y ago
The UK has the 'pay commercial users who are able to voluntarily reduce demand' system too, and that was part of the initial response to the frequency drop, I think. The problem is that the volume of demand you can get rid of this way isn't necessarily enough to be sufficient -- if you need to dump 10 or 20% of demand and especially if it needs to be spread evenly over the grid then it's going to be hard to do without disconnecting people involuntarily. I wouldn't be surprised if we could do better in the volume of demand we could reduce on demand, especially with smarter technology, though. At some point the question is whether the extra complexity and infrastructure is cost-effective to avoid once-a-decade events on the scale of this one, which probably 90% of the UK would never have noticed unless they read the news...

Slartie 7y ago

Sorry, but the electrical grid is not built like a Netflix server farm. Its explicit purpose is to provide reliability to the customer, so the customer will not have to invest into mitigations for network failures, because it is much cheaper, more economical and technically much more feasible to make these investments in the network than individually on every single customer location. The only exception are some very specific customers for which the provided amount of reliability is not enough - these need to increase the reliability by investing in additional failsafes, and are of course responsible for testing these by themselves (if they'd like to have random network outages performed, they are free to do so, because any consequences from these harsh tests are in their responsibility anyway, just as any consequences from not doing proper tests and thus having the safeguards potentially fail in an emergency). Netflix also would never intentionally break my FireTV box, which I use to watch Netflix, just to ensure that I have a backup in place (which I don't, because I don't care THAT much about Netflix availability). [edit] Now that I think about it...the purpose of the Netflix server farm is also to provide reliability to the customer, so it's actually comparable to a certain degree. It's just that the possible outcome of a mitigation failure in the Netflix server farm is limited to some people not watching Netflix for a few minutes, while the possible outcome of a power outage even for residential customers are usually far worse, let alone industrial or health customers. This allows Netflix to take the limited risk of intentionally breaking internals in their infrastructure, because the worst that can happen is them having to excuse themselves to some customers for a service interruption. Also, no one expects them to submit a report describing how it came to those interruptions, so they don't need to admit that they were the result of intentional self-inflicted disruptions (which sounds really crazy and irresponsible to pretty much all consumers out there). With electrical grid operators, the stakes are infinitely higher, and they are legally required to investigate any bigger problems and report details about their origins to both government authorities and customers. And you can be sure that "oh we just intentionally dropped 5% of the network to see if we get a full blackout or if our failsafes work" would not be an acceptable excuse to any of these two.

7952 7y ago

The best time to test a generator backup is when the grid power is still available. So that you can revert if the generator fails to start.

rossdavidh 7y ago

You would be setting up a Chaos Monkey (https://en.wikipedia.org/wiki/Chaos_engineering#Chaos_Monkey https://en.wikipedia.org/wiki/Chaos_engineering#Chaos_Monkey) on your electrical grid. The problem is, while these kinds of things are an awesomely good idea early in a system's life, they are very difficult to put in after it has become both complex and mission-critical.