13 ms·
System separation in the Continental Europe Synchronous Area on 8 January 2021
- GekkePrutser 6y agoAha so this is what caused that big frequency drop people had already reported. Interesting!
- londons_explore 6y agoIf you design an electricity grid to never partition, then you need to keep a substantial headroom of transmission equipment unused to prevent cascading failures leading to a partition in case of failure of just one or two sites. Instead, an electricity grid should be able to survive any partition like this while still keeping all frequencies within the nominal range. The total cost of such a system is lower in most cases (you need more idling generation capacity or droppable load, but less transmission capacity) The EU system here failed to do either here.
- frede 6y agoThe system survived the partitions, there was no widespread blackout. Further, the system is not organized by the EU, but the ENTSO-E and contains e.g. countries from North Afrika.
- cbmuser 6y agoThere was no blackout _yet_, but it was very close. According to the German Bundesnetzagentur (the equivalent of the FTC/EIA), the number of times where they have to intervene with the grid due to grid instability is constantly rising due to Germany shutting down nuclear and coal plants. > https://www.bundesnetzagentur.de/DE/Sachgebiete/ElektrizitaetundGas/Unternehmen_Institutionen/Versorgungssicherheit/Engpassmanagement/Redispatch/redispatch-node.html https://www.bundesnetzagentur.de/DE/Sachgebiete/Elektrizitae...
- phicoh 6y agoIt is a choice, spend a fortune on nuclear power that nobody wants in their own backyard, or deal with a less stable grid due to solar and wind energy.
- qayxc 6y agoSigh. It's not a dichotomy at all. It's perfectly possible to have a stable grid even with solar and wind. The problem is that neither nuclear nor coal power stations are load following and "base load" has lowered over the years (industry has moved to Asia, devices became more efficient, etc.). Stand-by power (like natural gas powered generators) hasn't been built up the way the way it should have been. Same goes for smart grid technologies and buffer storage; not to mention the maniacs (especially in South Germany) who basically protest everything - from nuclear power, to wind power, to required infrastructure like north-to-south high-voltage transmission lines. It's way too oversimplified to reduce the issue to just wind and solar.
- Dumbdo 6y ago> There was no blackout _yet_, but it was very close. Where do you get that from? None of the sources reported a close blackout, as far as I understood it there was a lot of emergency capacity left. We weren't even in the emergency frequency range, as the other commenter pointed out. Even the linked article just states that those interventions got more often after shutting down coal+nuclear, but it's not critical, it _only_ costs money to compensate the operators: https://de.wikipedia.org/wiki/Redispatch_(Stromnetz) https://de.wikipedia.org/wiki/Redispatch_(Stromnetz) It's probably much less money than all the nuclear subsidies.
- imtringued 6y agoAccording to this chart there has been an insignificant reduction in installed capacity. https://energy-charts.info/charts/installed_power/chart.htm?l=en&c=DE https://energy-charts.info/charts/installed_power/chart.htm?... Nuclear power went from 20GW to 10GW. Hard coal went from 28GW to 22GW. Ok, but gas went up from 23GW to 29GW. Brown coal stayed the same. Renewables went up by by 50GW for PV and 50GW for wind. Consider that wind often hits a 50% capacity factor. That is 25GW in additional power just from wind alone. Some of those plants may not be running continuously but they are still useful for emergency responses.
- jabiko 6y agoI'm not sure this counts as a failure. The load shedding mechanisms worked, supportive power generation was automatically activated and there was no widespread blackout. You are right that it appears like there was insufficient capacity at the north-west/south-east separation point and I guess that is going to be investigated but apart from that everything looks like it worked.
- zaarn 6y agoThe frequencies were in nominal range. The lowest was around 49.7, the highest about 50.6, both of which are within the allowed range of frequency (47.5 - 52.5 is the absolute emergency range, 49-51 is the nominal range, before you start capping power to or from customers). Romania had a power blackout for a bit, but everything worked as intended and the romanian grid blacking out didn't pull in the rest of the european grid. A partition between power grid is acceptable to preserve the largest amount of the grid remaining functional. Marrying the partitions together is bothersome but not something that takes forever (took 1 hour in this case). While 49.7 is very low, most of your devices will not notice and will be fine. Most industrial equipment is largely already coded to handle frequency shifts in favor of keeping the grid stable.
- gpvos 6y agoI understand that some equipment, such as consumer grade solar panels, already disconnects at 49.8, and one early analysis I read[0] said that that in fact exacerbated the problem somewhat on 8 January. [0] https://www.hoogspanningsnet.com/ https://www.hoogspanningsnet.com/ , entry of 08 januari 2021 - similar sentiments were expressed in articles linked from there
- zaarn 6y agoPower producers != Consumers for small devices like consumer grade solar panels, especially if the inverter is too cheap to handle a frequency excursion.
- deleted 6y ago[deleted]
- huhnmonster 6y agoI was wondering if someone could explain the countermeasures for such an event. Obviously, as the article states, producers are being shut off in the regions with surplus, while drains, who can afford to shut off, are shut off in the deficit regions. Is this an automatic process? Or is it more like someone from the company's energy provider calls them and tells them to shut off some devices? And is there not a potential problem, that if too many shut of at once, you now have a surplus again? Or is it coordinated by one single entity?
- deleted 6y ago[deleted]
- bo0tzz 6y agoIt is an automatic process, coordinated - as far as I'm aware - by two national energy grid providers (Switzerland and Germany) who have been selected for this role.
- fisian 6y agoThe response is automated and every system operator provides some ressources to react: https://en.wikipedia.org/wiki/Operating_reserve https://en.wikipedia.org/wiki/Operating_reserve
- Faaak 6y agoThese are called ancillary services. There are 3 levels: frequency control (for example dams), primary reverse (fast acting power, in less than 10s), and secondary reserve (slow acting: < 15min). Participants can either be positive (they consume more energy, for example PHES systems), or negative (they inject or consume less energy). All these ancillary services are paid (annual auction + per case). Nowadays, it's getting bigger with VPP: virtual power plants, which aggregate small loads (i.e. small ~1MW generators) in order to propose a bigger load to TSOs. It's all automatic If you're interested: https://www.swissgrid.ch/en/home/customers/ancillary-services.html https://www.swissgrid.ch/en/home/customers/ancillary-service...
- 6y ago
- brohee 6y agoInfosec slant: such investigations are made way easier by precise timestamping (everything happens in a few milliseconds), but the source of truth (GPS or other GNSS) is usually pretty easily spoofable if you only intend to move the time a few milliseconds. Galileo has a project addressing this (https://ec.europa.eu/transparency/regexpert/?do=groupDetail.groupMeetingDoc&docid=36951 https://ec.europa.eu/transparency/regexpert/?do=groupDetail....) but AFAIK it's not in the signal yet. And once it is, it will take years for the devices doing the satellite time to PTP to be replaced/updated.
- traceroute66 6y agoSurely this is only true to a minimal extent in world where you can already buy timeservers that support multiple independent time sources out of the box (multiple satellite Galileo,GPS etc. + terrestrial radio e.g. DCF77). IIRC if you're dead serious and money is no object, all the major national time labs also allow you to run a private leased line to their facility for direct checks.
- brohee 6y agoAs of now all GNSS constellation are somewhat spoofable, especially if you only want to drift a clock a few milliseconds late... And a lot of timeservers are still GPS only so the attack is pretty easy to perform. The private line idea is maybe doable (but which protocol? NTP doesn't offer the needed precision), but as all substations would need one, my guess is that eventually money would be an object...
- michaelt 6y ago> (but which protocol? NTP doesn't offer the needed precision) You might enjoy reading the documentation for NPLTime [1] which uses PTP over telecom fibre networks. Of course, their target market is banks with record keeping requirements so it probably costs a fortune. [1] https://www.npl.co.uk/npltime https://www.npl.co.uk/npltime
- jasonjayr 6y agoPreviously, from the 8th: https://news.ycombinator.com/item?id=25685646 https://news.ycombinator.com/item?id=25685646
- contravariant 6y agoSeems like an excellent example that preventing a single point of failure means nothing if your system doesn't have the extra capacity to handle a single failure.
- nabla9 6y agoThere was multiple failures, not single. The grid was separated but it didn't cause more power outages.
- LargoLasskhyfv 6y agoYou know what I'm missing? The almost realtime map from ucte.org which updated every 3 to 5 seconds on my Pentium3 933 with only 512MB Ram in the browser. I think it needed some plugin, either Java or Flash, but you could zoom in or out, either to the whole of Europe, or down to the substations in the cities and towns. It was like google maps or similar. AND it had all the transmission lines, power plants, and substations. And color coded lines, with KW/MW/GW and arrows indicating the direction of energy flow on them. Anybody could view it without having an account there. At least I did, from time to time. This way of viewing it in realtime is now gone, or at least not accessible to the general public anymore. Does anybody know some equivalent? Sites with bargraphs and charts need not apply.
- Reason077 6y agoENTSO-E is the successor organisation to UCTE [1], with its functions transferred to ENTSO-E in 2009. So if anyone has the data, it would be them. Interesting that the ucte.org website still seems to be online and frozen in time! [1] https://www.entsoe.eu/news-events/former-associations/#union-for-the-coordination-of-the-transmission-of-electricity-ucte https://www.entsoe.eu/news-events/former-associations/#union...
- andreasley 6y agoWhile ENTSO-E does have a map [1], it's not as nice as the one described above. [1] https://www.entsoe.eu/data/map/ https://www.entsoe.eu/data/map/
- LargoLasskhyfv 6y agoYes, does not compare, unfortunately.
- noobermin 6y agoI hate to commit red-herring but this is another example of how computing is worse today in some ways.
- AnssiH 6y agoAFAIK the continental synchronous area frequency is long-term-synchronized to atomic time (i.e. the cumulative grid time must stay within X seconds of the reference clock). Looking at the (partial) graph suggests that the south-east part ran faster for the entire duration of the disconnection. Does this mean grid-synced clocks in the south-east part are now permanently ahead of the atomic reference, or are they planning a mitigation (which I assume would have to mean disconnecting again and running the SE part slower for a bit)?
- gspr 6y ago> Does this mean grid-synced clocks in the south-east part are now permanently ahead of the atomic reference, or are they planning a mitigation (which I assume would have to mean disconnecting again and running the SE part slower for a bit)? That's the standard procedure. Maintaining an exact frequency can be hard, but counting cycles is easy. So the frequency will slew to slowly compensate back. Edit: Here's a Tom Scott video on a UK–Continental divergence that happened a while back: https://www.youtube.com/watch?v=bij-JjzCa7o https://www.youtube.com/watch?v=bij-JjzCa7o
- garaetjjte 6y agoUK is connected through DC lines, so frequency is independent. In this case they would have to separate the grid again to be able to correct this.
- tyho 6y agoHow does this affect timekeeping? Normally, if there is a drop in grid frequency, a small increase will be planned later to maintain a long term average frequency of 50Hz so that clocks that keep time by grid frequency stay accurate [0]. The network split seems to have made this impossible, during the split, the cycle counts for the two regions diverged, and the split ended before this was reconciled. Will people in one of the regions have to adjust their clocks? [0]: https://en.wikipedia.org/wiki/Wide_area_synchronous_grid#Timekeeping https://en.wikipedia.org/wiki/Wide_area_synchronous_grid#Tim...
- i_am_proteus 6y agoMy suspicion is that NTP (and GPS) have generally superseded the synchronous grid for timekeeping. It has for every application that I know of. Simultaneously, I suspect some systems still do rely on grid time.
- nottorp 6y agoComputers (or anything containing a proper computer) don't care about grid frequency for time keeping. On the consumer side, your microwave, coffee machine and even bedside clock will. Not sure on the industrial side.
- read_if_gay_ 6y ago> your microwave, coffee machine and even bedside clock will. I guess a ~0.5% deviation for a couple minutes is going to be tolerable here.
- alex_duf 6y agoI wonder what are the benefits of keeping the grid synchronous? Would it be possible to have multiple smaller grids, still interconnected, but without being kept in sync? I'm not sure if what I'm saying is possible or efficient, but converting AC to DC, transmitting the energy, then converting DC to AC so the frequency becomes irrelevant.
- gspr 6y agoFor one, it's used for timekeeping.
- filmor 6y agoThat's more of a side-effect, not the goal of a synchronous grid.
- tsar_nikolai 6y ago> I wonder what are the benefits of keeping the grid synchronous? > Wide area synchronous networks improve reliability and permit the pooling of resources. Also, they can level out the load, which reduces the required generating capacity, allow more environmentally-friendly power to be employed; and allow more diverse power generation schemes and permit economies of scale. [0] [0] https://en.m.wikipedia.org/wiki/Wide_area_synchronous_grid https://en.m.wikipedia.org/wiki/Wide_area_synchronous_grid
- calaphos 6y agoIt is absolutely possible and exactly how connections between the different grids are done, e.g. between the power grids of North America or between the European grid and Russia. HVDC of course has the disadvantage of extra conversion equipment. HV circuit breakers are also significantly more complex as an arc will form as long as current is flowing. With AC this happens at the zero automatically, nothing like this with DC.
- asmithmd1 6y agoYou can see on this map the purple DC lines between England and France and Ireland https://www.entsoe.eu/data/map/ https://www.entsoe.eu/data/map/ I am amazed the grid goes all the way to Iraq
- cpach 6y agoRelevant Wikipedia article: https://en.wikipedia.org/wiki/Synchronous_grid_of_Continental_Europe https://en.wikipedia.org/wiki/Synchronous_grid_of_Continenta... Must admit I’ve never heard of this before :-)
- m463 6y agoFunny how the folks on this site will still wade into an article even with a completely unrecognizable (yet curious) title like this one.
- deleted 6y ago[deleted]
- kuter 6y agoThere was a incident in Turkey (31 march 2015) that caused a country wide blackout. From what I understand the west side and the east side loose sync. The west side had under supply and the east side had over supply.
- reitanqild 6y agoWhat puzzles me is why everyone talks about clocks..! Clock synchronisation is a nice side effect, but what amazes me is how it is possible to keep this thing swinging somewhat synchronously year after year, and how huge chunks of it doesn't crash and burn when events like this happens. (I have worked as a consultant for a power supplier, I was more puzzled afterwards : )
- LargoLasskhyfv 6y agoTell us more about that :)
- reitanqild 6y agoI cannot really but I have put my lack of knowledge on public display here for someone else to help me and you : https://news.ycombinator.com/item?id=25917037 https://news.ycombinator.com/item?id=25917037 : )
- LargoLasskhyfv 6y agoThanks. I've already read that ;) But let me say this. About 10 to 15 years ago I've been more interested in transmission lines because of "Energiewende" and such, came across HVDC, how that would enable smart grids, possibly protection against solar/geomagnetic storms because of faster switching times, and so on. I looked into how they built those HVDC-lines in China, how that is not new at all, but eased by modern solid-state power semiconductors, and so on. And looked further into patents, modulation schemes, even many 'papers'(pdf) and also very technical books(skimmed those only). Anyway, what I learned was that the grid has many very nonlinear phenomena which are extremely hard to model and predict, because of the dynamics. Much more complicated than the simplistic answers some here have given you. So...relax? :-)
- littlestymaar 6y ago> what amazes me is how it is possible to keep this thing swinging somewhat synchronously year after year There's nothing to do to keep them in sync: every single[1] alternator connected to the electric grid is rotating at the same speed, which is driven by the frequency of the electricity. This is a physical effect. The big challenge isn't to keep things in sync, it's to keep the supply equals to the demand. (when we don't, the frequency goes up or down depending on which is higher than the other) [1]: in thermal stations only, wind turbines are not directly connected to the grid because their rotation speed depends on the wind…
- liversage 6y agoSomething similar happened in 2018. However, it was not a technical problem but a conflict between the power grid operators in Serbia and Kosovo. https://www.dw.com/en/clocks-in-europe-are-running-late-because-of-the-kosovo-conflict/a-42867061 https://www.dw.com/en/clocks-in-europe-are-running-late-beca... I believe that if you adjusted your clock during the conflict you had to adjust it again when it was resolved as the resolution was to increase the frequency for a period to reverse the loss of frequency.