11 ms·

This sounds big enough to require a black start. Unfortunately, those are slow and difficult. If an entire nation trips offline then every generator station di

This sounds big enough to require a black start. Unfortunately, those are slow and difficult. If an entire nation trips offline then every generator station disconnects itself from the grid and the grid itself snaps apart into islands. To bring it back you have to disconnect consumer loads and then re-energize a small set of plants that have dedicated black start capability. Thermal plants require energy to start up and renewables require external sources of inertia for frequency stabilization, so this usually requires turning on a small diesel generator that creates enough power to bootstrap a bigger generator and so on up until there's enough electricity to start the plant itself. With that back online the power from it can be used to re-energize other plants that lack black start capability in a chain until you have a series of isolated islands. Those islands then have to be synchronized and reconnected, whilst simultaneously bringing load online in large blocks. The whole thing is planned for, but you can't really rehearse for it. During a black start the grid is highly unstable. If something goes wrong then it can trip out again during the restart, sending you back to the beginning. It's especially likely if the original blackout caused undetected equipment damage, or if it was caused by such damage. In the UK contingency planning assumes a black start could take up to 72 hours, although if things go well it would be faster. It's one reason it's a good idea to always have some cash at home. Edit: There's a press release about a 2016 black start drill in Spain/Portugal here: https://www.ree.es/en/press-office/press-release/2016/11/spain-portugal-and-france-conduct-a-joint-response-drill-to-a-general-blackout https://www.ree.es/en/press-office/press-release/2016/11/spa...

leomca 2y ago
It seems Spain lost 15GW of load, but is still running 10GW of load: https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/show?name=&defaultValue=false&viewType=GRAPH&areaType=BZN&atch=false&dateTime.dateTime=28.04.2025+00:00|CET|DAY&biddingZone.values=CTY|10YES-REE------0!BZN|10YES-REE------0&dateTime.timezone=CET_CEST&dateTime.timezone_input=CET+(UTC+1)+/+CEST+(UTC+2) https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/sh... Would this suggest the grid hasn't snapped apart, or is it just not possible to tell from the data? Coal, pumped hydro, and nuclear generation all went to 0 around the same time, but presumably that's those sources being disconnected from the grid to balance demand? https://transparency.entsoe.eu/generation/r2/actualGenerationPerProductionType/show?name=&defaultValue=false&viewType=TABLE&areaType=BZN&atch=false&datepicker-day-offset-select-dv-date-from_input=D&dateTime.dateTime=28.04.2025+00:00|CET|DAYTIMERANGE&dateTime.endDateTime=28.04.2025+00:00|CET|DAYTIMERANGE&area.values=CTY|10YES-REE------0!BZN|10YES-REE------0&productionType.values=B01&productionType.values=B02&productionType.values=B03&productionType.values=B04&productionType.values=B05&productionType.values=B06&productionType.values=B07&productionType.values=B08&productionType.values=B09&productionType.values=B10&productionType.values=B11&productionType.values=B12&productionType.values=B13&productionType.values=B14&productionType.values=B20&productionType.values=B15&productionType.values=B16&productionType.values=B17&productionType.values=B18&productionType.values=B19&dateTime.timezone=CET_CEST&dateTime.timezone_input=CET+(UTC+1)+/+CEST+(UTC+2) https://transparency.entsoe.eu/generation/r2/actualGeneratio...
- mike_hearn 2y ago
  They're definitely doing a black start: https://x.com/RedElectricaREE/status/1916818043235164267 https://x.com/RedElectricaREE/status/1916818043235164267 We are beginning to recover power in the north and south of the peninsula, which is key to gradually addressing the electricity supply. This process involves the gradual energization of the transmission grid as the generating units are connected. I see load dropping to zero on that graph, or rather, load data disappears an hour ago. If the grid frequency goes too far out of range then power stations trip automatically, it's not an explicit decision anyone takes and it doesn't balance load, quite the opposite. A station tripping makes the problem worse as the frequency drops even further as the load gets shared between the remaining stations, which is why grids experience cascading failure. The disconnection into islands is a defense mechanism designed to stop equipment being too badly damaged and to isolate the outage.
  - leomca 2y ago
    Interesting, but in terms of load I think think the data may just be delayed by ~1 hour. Switching to UTC, to avoid timezone confusion, it's currently 13:10: Last actual load value for Spain at 12:15: https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/show?name=&defaultValue=false&viewType=TABLE&areaType=BZN&atch=false&dateTime.dateTime=28.04.2025+00:00|UTC|DAY&biddingZone.values=CTY|10YES-REE------0!BZN|10YES-REE------0&dateTime.timezone=UTC&dateTime.timezone_input=UTC https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/sh... Last actual load value for France at 12:00: https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/show?name=&defaultValue=false&viewType=TABLE&areaType=BZN&atch=false&dateTime.dateTime=28.04.2025+00:00|UTC|DAY&biddingZone.values=CTY|10YFR-RTE------C!BZN|10YFR-RTE------C&dateTime.timezone=UTC&dateTime.timezone_input=UTC https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/sh...
    - mike_hearn 2y ago
      It may also be a measurement artifact. Looking at the generation by type page, https://transparency.entsoe.eu/generation/r2/actualGenerationPerProductionType/show https://transparency.entsoe.eu/generation/r2/actualGeneratio... Everything dropped to zero except wind and solar, which took huge hits but not to zero. I expect those have been disconnected too, as they cannot transmit to the grid without enough thermal plant capacity being online, but if the measurement at some plants of how much they're generating doesn't take into account whether or not they were disconnected upstream they may still be reporting themselves as generating. You can't easily turn off a solar plant after all, just unplug it. Either that, or they're measuring generation and load that's not on the grid at all.
martinald 2y ago
Interestingly it seems that the black start drill is considering a smaller zone of impact than what has happened here. Also I suspect there is far more renewables on the grid now than in 2016. This is potentially the first real black start of a grid with high renewable (solar/wind) penetration that I am aware of. Black starts with grids like this I imagine are much more technically challenging because you have generation coming on the grid (or not coming on) that you don't expect and you have to hope all the equipment is working correctly on "(semi)-distributed" generation assets which probably don't have the same level of technical oversight that a major gas/coal/nuclear/hydro plant does. I put in another comment about the 2019 outage which was happened because a trip on a 400kV line caused a giant offshore wind farm to trip because its voltage regulator detected a problem it shouldn't have tripped the entire wind output over. Eg: if you are doing a black start and then suddenly a bunch of smallish ~10MW solar farms start producing and feeding back in "automatically", you could then cause another trip because there isn't enough load for that. Same with rooftop solar.
- genewitch 2y ago
  Grid tied solar won't put power into the grid when the grid is down. It's the one reason I didn't grid tie. Non tied solar won't affect the grid at all. So this is a non-issue. Grid tie requires the grid to tie to, otherwise it can't synchronize. So it stays disconnected.
  - martinald 2y ago
    Youre missing my point. I know that; I mean if you are restarting the grid and say you have a segment you think has 5MW of load that will come online, but you connect it and a minute or two later suddenly 2MW of grid tied solar detects the grid and starts exporting and you now have 3MW of load it is going to make it much more tricky to balance the restart. I'm not sure how much of a problem this is in reality but it seems to me restarting a grid is made much more tricky when you have millions of generation assets with no control over.
    - christkv 2y ago
qyckudnefDi5 2y ago
This guy grids!
chippiewill 2y ago
The frequency aspect of a black start is presumably a bit easier in Europe because there's an interconnected synchronous grid so they can bootstrap it from France essentially. It's far more problematic for the UK because all the interconnects are DC.
- padjo 2y ago
  I was recently told by an electrical engineering lecturer that the black start plan here in Ireland is to use the DC interconnectors with the UK to provide startup power to a synchronous generator.
  - ajmurmann 2y ago
    Did he mention the UK's plan in case they have a full blackout?
    - pjc50 2y ago
      https://www.nationalgrid.com/sites/default/files/documents/High%20Level%20Black%20Start%20Strategy.pdf https://www.nationalgrid.com/sites/default/files/documents/H... (2017, probably current)
      - InsomniacL 2y ago
        "it is prudent that the cost of maintaining Black Start Capability should demonstrate value to the end consumer" That sounds like someone explaining why the solution is so bad, before describing what the solution is.
        pbhjpbhj 2y ago
        To me it sounds like an energy company attempting to excuse lack of spending on infrastructure whilst paying out millions to C-suite in bonuses and millions more to shareholders whilst arguing prices have to rise because they can't afford to spend on infrastructure...
celticninja 2y ago
cash is all well and good but if the tills dont work most stores wont be able to serve you. corner shops, vape shops and barbers will be ok though
- genewitch 2y ago
  Stores with tills and freezers etc will have power for the tills but the backing network for payments probably won't be up. That's the concern with being cashless. They can accept cashxl, but no one has any.
- petertodd 2y ago
  The tills have keys to manually open them, and you can just record transactions on pen-and-paper if needed and enter them later. I've seen plenty of businesses do exactly that during power outages. Not as fast. But totally doable.
- torcete 2y ago
  I was able to buy some groceries and pay with card. The tills had a battery backup and the network infrastructure that supports card payments was apparently working. That said, lots of people hit the cafés and had to resource to cash payments. There was also lots of people buying bottled water at the shops. So basically, you could divide people in two groups. Those that took it like an extra Sunday, and those that took it like the beginning of a war or something :-D
  - diggan 2y ago
    Most people around here seemed to be balanced. Yes, a lot of people outside at bars/cafes enjoying a spare Sunday, but also preparing quietly and slowly in case it takes longer time. All the stores ran out of batteries, radios and such, at least the ones we visited, and I only managed to make one card payment around 13:30 sometime, after that internet stopped working 100% until the night. Almost no super markets were open around here, maybe one or two who had generators it seems.
floatrock 2y ago
Practical Engineering did a really great video a few years ago on why black starts are hard, complete with a tabletop demo about the physics of synchronizing large spinning generators: https://www.youtube.com/watch?v=uOSnQM1Zu4w https://www.youtube.com/watch?v=uOSnQM1Zu4w
apexalpha 2y ago
Would this be relevant here? Spain is connected to the French grid, and probably also Morocco. The entire EU runs on one synchronised grid so from that perspective a single 'province' went offline, not the grid.
- crmd 2y ago
  It would be essentially the same thing as a grid black start, except that the first breaker to close has the European grid on its primary side, instead of a freshly started generator under your control. The complex process of configuring the transmission network to bring grid power to each power plant in succession is the same.
- lysace 2y ago
  The continental part of the EU runs on one synchronized grid. The Nordics (except half of Denmark and, uh, Iceland, Greenland etc) runs on a separate synchronized grid. https://en.wikipedia.org/wiki/Continental_Europe_Synchronous_Area#/media/File:European_electricity_grid.svg https://en.wikipedia.org/wiki/Continental_Europe_Synchronous...
- jameshart 2y ago
  Re: probably also Morocco: yes, there is an interconnect across the Strait of Gibraltar https://en.wikipedia.org/wiki/Spain-Morocco_interconnection https://en.wikipedia.org/wiki/Spain-Morocco_interconnection
  - dom96 2y ago
    Speaking of Gibraltar, it's interesting that its electricity grid isn't connected to Spain/EU's grid. So they are completely unaffected.
mlsu 2y ago
Does solar power make this process easier or harder? I know that with thermal plants you have a spinning mass that you have to synchronize, and phase shift is used to assess how hard the plant is working (and whether to trip a disconnect as we see here) But with solar, how is the synchronization provided? In like a giant buck? Or in software somehow? Does the phase shift matter as much as in the electromechanical systems? My intuition is that solar would make the grid harder to keep stable (smaller mass spinning in sync) but also may offer more knobs to control things (big DC source that you can toggle on/off instantly.. as long as sun is out). But I don’t actually know.
- mike_hearn 2y ago
  Most solar and wind plants follow the inertial lead of the thermal plants. They can't synchronize without enough thermal generation being online. Supposedly there are efforts to change that, I don't know enough about grid engineering to say how far along that might be in Spain.
- pjc50 2y ago
  Mike_hearn's comment was grey but was correct: phase following is indeed done through software in the inverter. Phase matching is still required, wherever the phase difference is not zero there is a deadweight loss of power as heat. Currently the main driver of battery deployments is not so much energy price time arbitrage as "fast frequency fresponse": you can get paid for providing battery stabilization to the grid. (for the UK not Spain: https://www.axle.energy/blog/frequency https://www.axle.energy/blog/frequency )
  - mlsu 2y ago
    Oh that is fascinating. So if you have a smarter solar panel, or a smart battery, you can stabilize the grid. I’m assuming that all of the traditional software complexity things in distributed systems apply here: you want something a little bit smart, to gain efficiency benefits, but not too smart, to gain robustness benefits. My intuition is that bringing the market into it at small timescales probably greatly increases the efficiency significantly but at the cost of robustness (California learned this “the hard way” with Enron) > Phase matching is still required, wherever the phase difference is not zero there is a deadweight loss of power as heat If the electronic controller is “ahead of” (leading) the grid, then that heat would come from the solar plant; if it is “behind” (following) then that heat comes from the grid. Is that right? And likely, solar plants opted for the simplest thing, which is to always follow, that way they never need to worry about managing the heat or stability or any of it. I wonder if the simplest thing would be for large solar plants to just have a gigantic flywheel on site that could be brought up via diesel generators at night…
bob1029 2y ago
> Thermal plants require energy to start up It's not just about the power. System components cannot be brought to operating temperatures, speeds and pressures faster than mechanical tolerances allow. If a thermal plant is cold & dark, it can take days to ramp it to full production.
- pfdietz 2y ago
  That's true of some kinds of thermal generators, but not all. Simple cycle gas turbines can come up very quickly (think jet engines). Or your car's engine.
  - bob1029 2y ago
    > Simple cycle gas turbines can come up very quickly In some cases yes. Modern combined cycle plants can take as little as 30 minutes to ramp to full output. Older designs can take upward of 4-6 hours. If you have steam as an indirection, that's when things take a really long time. Natural gas turbines are a more direct cycle.
    - pfdietz 2y ago
      Combined cycle turbines do have a steam component (that's where the word "combined" comes from). The waste heat from the combustion turbine front end is used to make steam in the back end.
      - bob1029 2y ago
        Combined cycle turbines are not primarily powered by steam. It is a secondary consequence of their operation. This contrasts with nuclear and coal plants where steam is the prime mover.
        genewitch 2y ago
        That's what they said, though
        2y ago
Vox_Leone 2y ago
The fewer resources we dedicate to grid resilience and modernization, the harder black starts become. And as grids get more complex and interdependent, recovering from total failure becomes exponentially harder. A rare but sobering opportunity to reflect on something we usually take for granted: electricity. We live in societies where everything depends on the grid — from logistics and healthcare to communications and financial systems. And yet, public awareness of the infrastructure behind it is shockingly low. We tend to notice the power grid only when it breaks. We’ve neglected it for decades. In many regions, burying power lines is dismissed as “too expensive.” But compare that cost to the consequences of grid collapse in extreme weather, cyberattacks, or even solar storms — the stakes are existential. High-impact, low-frequency events are easy to ignore until they’re not.
- roughly 2y ago
  Yeah, this is the turkey’s dilemma - life on a farm is a lot better than life in the wild for 51 out of the 52 weeks of the year. Most of our modern economy and systems are built to reduce redundancy and buffers - ever since the era of “just in time” manufacturing, we’ve done our best to strip out any “fat” from our systems to reduce costs. Consequently, any time we face anything but the most idealized conditions, the whole system collapses. The problem is that, culturally, we’re extremely short-termist- normally I’d take this occasion to dunk on MBAs, and they deserve it, but broadly as a people we’re bad at recognizing just how far down the road you need to kick a can so you’re not the one who has to deal with it next time and we’ve gotten pretty lazy about actually doing the work required to build something durable.
  - Slava_Propanei 2y ago
    [dead]
  - maxerickson 2y ago
    What are some examples of modern system collapse? We've had substantial disruptions, but they've not been particularly irrecoverable or sustained.
dv_dt 2y ago
I would think that renewable infrastructure could be the fix, at least if you start installing larger battery capacity to meet renewable store and usage shifts, the grid essentially is installing the resources that can also be used to respond to & contain sudden source losses and prevent cascades.
oilman 2y ago
In another life I worked as an engineer commissioning oil rigs and I’ve seen how tricky even a small-scale black start can be. On a rig, we simulate total power loss and have to hand-crank a tiny air compressor just to start a small emergency generator, which then powers the compressors needed to fire up the big ~7MW main generators. It's a delicate chain reaction — and that's just for one isolated platform. A full grid black start is orders of magnitude more complex. You’re not just reviving one machine — you’re trying to bring back entire islands of infrastructure, synchronize them perfectly, and pray nothing trips out along the way. Watching a rig wake up is impressive. Restarting a whole country’s grid is heroic.
- zdragnar 2y ago
  I can appreciate the ability to revert to hand cranking an air compressor, yet I can't help but feel that the 99.99% of events, you'd be better served with keeping a two stroke gas engine ready to go. Air compressors tend to have parts just as or more vulnerable to environmental factors, and you get a lot more power for less elbow grease out of a two stroke.
  - Symbiote 2y ago
    A power station can start a decently large generator with batteries. Maybe there are other concerns for an oil rig.
    - tonyarkles 2y ago
      Batteries are great when they have charge. What happens if the generator doesn't want to start the first, second, and third time? How many start attempts do you get before the batteries are dead? The hand-pumped air compressor is the tool of last resort. You can try an engine start if there's someone there who's able to pump it. You don't have to worry about how much charge is left in your batteries or whether or not the gasoline for the 2-stroke pump engine has gone stale. It's the tool that you use as an alternative to "well, the batteries are dead too, guess we're not going to start the engine tonight... let's call the helicopters and abandon ship"
egorfine 2y ago
Ukraine can share some experience with forced black starts.
- petertodd 2y ago
  Ukraine is interesting in this context because there are so many generators. In the richer parts of Odesa I've even found it hard sometimes to tell whether or not the grid power is on as literally every single building had sufficient backup generators to keep the lights on (also, many big businesses seemed to run their generators even when the power was on, I presume as a civic-minded way to add generation capacity overall and avoid interruptions when the power flips on and off).
  - egorfine 2y ago
    Generators are not in sync with the grid though. And that's the hardest part. Here's why generators were running here despite the grid being available. A generator has a very short lifetime and in order to prolong it, some owners learned to run those in the very optimal schedule. Which sometimes requires a minimum amount of time to run in a single cycle. Thus if you started it you are committed to run for X hours.
    - petertodd 2y ago
      > Generators are not in sync with the grid though. And that's the hardest part. I know. What's interesting re: Ukraine is because there are so many generators there are more options to getting power sufficiently restored for normal life than just rushing to restore the entire grid. > Thus if you started it you are committed to run for X hours. I'm skeptical this is the main reason. While fast starting a diesel generator is hard on it, there are other, slower, ways to start big diesel generators with minimal impact on lifespan. The blackouts in Ukraine are almost all on a schedule, so big buildings with dedicated staff and expensive generators can and do startup their generators in advance (I've personally heard this happen on occasion - big generators spooling up multiple minutes in advance of the scheduled blackout). Also, it makes sense to turn on generators in advance anyway: gives you a chance to diagnose any issues.
gwbas1c 2y ago
Spain is part of the largest interconnected grid in the world: https://en.wikipedia.org/wiki/Continental_Europe_Synchronous_Area#/media/File:European_electricity_grid.svg https://en.wikipedia.org/wiki/Continental_Europe_Synchronous... Does this really qualify as "black start" when they can rely on the bigger EU grid?
- throwup238 2y ago
  Yes because they have to bring it all back up in phases so that they only face the load spike* from one interconnect at a time, which can take some time and can fail if there’s unknown damage like the GP said. It really depends on the region though because almost all large hydroelectric dams are designed to be primary black-start sources to restore interconnects and get other power plants back up quickly in phase with the dam. i.e. in the US 40% of the country has them so it’s relatively easy to do. The hardest part is usually the messy human coordination bit because none of this stuff is automated (or possible even automatable). * the load spike from everyone’s motors and compressors booting up at the same time
  - pbhjpbhj 2y ago
    Presumably emergency phone comms still work(?) so they could issue instructions to do a phased (heh!) restart to avoid every fridge/air conditioner/whatever restarting at once. Not sure how successful that would be however.
    - throwup238 2y ago
      There’s usually two parallel processes going on during a black-start: spinning the power plants back up to get them synchronized to the grid, and getting power back out to all the consumers. Power plants have breakers that disconnect them from consumers which they keep disconnected until independent connections to other power plants allow them to spin up their massive turbines and synchronize them to the grid. There’s also tons of other downstream breakers at substations which will be in various states of functionality. The power plants with direct connections have hard lines and black-start procedures that get power out to the most important customers like telecom infrastructure, which provides the rest of the comms. In a real world full restart it’s going to mean organizing workers at many substations to babysit old infrastructure so cellular is pretty much mandatory.
larusso 2y ago
I actually have no real clue how and where Spain/Portugal is connected to rest of Europe but could they also restart North to south with help from the grid from France?
kennysmoothx 2y ago
This is literally like Factorio. If your Factory uses too much power, theres not enough energy to run the power plants generation, which decreases your power production. Death spiraling until theres no power. You have to disconnect the factory, and independently power your power plants back up until you have enough energy production to connect your factory up again.
- eddd-ddde 2y ago
  In my server I hooked up a sound alarm to a set of capacitors. Too low of a charge indicates higher power consumption than production, allowing you to unplug certain low priority loads. I also have some emergency coal generators ready to go at the flick of a switch if needed. Man I need to go play some more.
- baq 2y ago
  They should allow you to build automatic rate-of-change-of-power-generation breakers...
  - abdullahkhalids 2y ago
    Factorio has logic circuits which allow you to do this.
- shagie 2y ago
  Capacitor circuit network warning for alarm for "main grid is drawing on this bank - bad things may happen if capacity is not increased." Another "trick" is those burner inserters are black start capable. They can pick up fuel and feed themselves to keep running without an electrical network. I also tend to put Schmitt triggers in low priority areas. They've got a battery on the main grid next to them and if the battery drops below 50% power they remain off until it goes back above 75% power.
  - tux3 2y ago
    Can't wait for a Factorio mod that simulates the grid frequency and damages equipment when it drifts too far!
belter 2y ago
One of the Administrators of the REN, the Portuguese electric supplier is currently giving a press conference. Confirmed they are in scenario of restart from black start. - Cause of event not known yet. - They noticed power oscillations from the Spanish grid that tripped safety mechanisms in the Portuguese grid. At the time, due to the cheaper prices, the Portuguese grid was in a state of importing electricity from Spain. - They are bringing up multiple power systems and the Portuguese grid is able to supply 100% of needs if required. It was not configured in such a state at the moment of event. - They had to restart the black start more than once, since while starting, noticed instabilities in some sectors that forced them to restart the process. - Time for full recovery unknown at this time, but it will take at least 24 hours.
lastdong 2y ago
Can’t upvote enough! Thank you!
beefnugs 2y ago
And if you are a mere mortal in this world, play Factorio : Space age expansion on the planet Aquilo. To learn the precious importance of reliable multi stage power bootstrapping
anovikov 2y ago
Power never went out in a country completely. At the lowest point consumption was ~40% of normal for that time of day. Ukraine went through many black starts in the first winter of Russian strikes against energy. I guess they built a skill of recovering it quickly enough that it started happening faster and easier every next time.
Fire-Dragon-DoL 2y ago
Oh damn, I thought this was a problem only in Dyson Sphere Program, lol
sgt101 2y ago
I did a project where we predicted transformer failure (they blow up!) from changes in the oil that they have in them (its insulative properties suddenly change). This was 30 years ago so it's all a bit fuzzy, but the one thing that really stuck with me was the story that the SME that we were working with had about the UK grid teetering on the edge of failure during the "great storm" of 1987. His telling was that they really were unsure if they would get it back at all!
bsder 2y ago
>It's one reason it's a good idea to always have some cash at home. Most places are so dependent upon electricity that they can't even take cash during a blackout. And they don't even have the mechanical machines to take a credit card imprint anymore.
- brewdad 2y ago
  The last of my raised number credit cards went away last month. Those old machines will only get a blank rectangle from me. Sad, because I did actually use one of those about 3 years ago when a rural gas station had power but no network.
  - folmar 2y ago
    Network is not required for mag-strip or emv operations, but the terminal has to be configured accordingly (and has to have some way to send batch of transactions for settling). It's less common now, but fully supported.
    - TrueSlacker0 2y ago
      If the network comes back up within 24hrs.
isolli 2y ago
What does synchronization mean in this context, and why is it needed?
- seabrookmx 2y ago
  The bulk of the power grid is alternating current (AC), and the frequency of the resulting sine wave needs to be synchronized with the other parts of the grid it is connected to.
- Tempest1981 2y ago
  The grid uses AC (not DC), running at 50 Hz (cycles per second). So the voltage is going up and down at that frequency, in a sine-wave pattern. If you try to connect another generator to the grid, it needs to be at the same point (phase) in the sine-wave cycle, so that its power contribution is added, not subtracted. If it's not in sync, huge currents can flow, causing damage. Sort of like connecting jumper cables backwards.
hnthrowaway0315 2y ago
I wonder if someone could build a realistic scenario into a game -- let's say some sort of smaller scale black start for, say, a space station. And throw in a unknown computer architecture for the in-game computers so that players need to RTFM to figure out how it works.
- Xorakios 2y ago
  Awesome idea!
- sterlind 2y ago
  make it so! the fun of reverse engineering systems I don't understand to fix them is half the reason I joined this industry.
- dxuh 2y ago
  I tried to make something like this (not power specifically) for a game jam once: https://pfirsich.itch.io/arbitrary-complexity https://pfirsich.itch.io/arbitrary-complexity I took down the servers though, so you probably can't easily try it. I don't know if I added a way to configure the lobby server. I should have! It's open source though. And there is a video about that thing on my YouTube: https://www.youtube.com/watch?v=6TPgfa7LbiI https://www.youtube.com/watch?v=6TPgfa7LbiI The game is bad and nothing of what we planned on doing actually made it into the game. The video is long and boring too. But maybe someone finds this cool and is inspired by this and makes a game like this. The first 15 minutes of the game were actually about getting the ship moving, first by reading the manuals of half a dozen different ship systems and then following some procedure outlined in those manuals (parts of which were simply incorrect), maybe having to do some things in sync with your other players and stuff like that. I think it would have been cool to add multiple reactors and start them up in sync and stuff. The different ship systems were actually Lua programs that interacted via a message bus. So kind of a unknown computer architecture?
  - hnthrowaway0315 2y ago
    Wow this really looks interesting.
bill3478 2y ago
A true black start has several factors (which make it difficult and notable): 1. The grid has to fully collapse with no possibility of being rescued by interconnection 2. As a result, a generation asset has to be started without external power or a grid frequency to synch to 3. An asset capable of this is usually a small one connected to a lower voltage network that has to then backfeed the higher voltage one 4. Due to the difficulty of balancing supply/demand during the process, the frequency can fluctuate violently with a high risk of tripping the system offline again None of this applies in yesterday's case: The rest of the European synchronous grid is working just fine. News reports stated Spain restored power by reconnecting to France and Morocco. By reestablishing the HV network first, they can directly restart the largest generation asset with normal procedures. As they bring more and more load or generation online, there's little risk of big frequency fluctuations because the wider grid can absorb that.
fredo2025 2y ago
It’s not just restarting it that’s difficult. There are the side effects like what happened in NY: https://en.wikipedia.org/wiki/New_York_City_blackout_of_1977 https://en.wikipedia.org/wiki/New_York_City_blackout_of_1977 > have some cash at home For maybe the first 24 hours at a grocery store, and then not so sure. Would your neighbors sell you supplies and food? Maybe not? And so many places now depend on cashless transactions and doubtful they have pen, paper, lockbox, and safe as a contingency plan.
greggsy 2y ago
Sounds like my last Satisfactory build.
bigbacaloa 2y ago
[dead]
fifteen1506 2y ago
Black Start achieved! Not at 100% yet. Apparently we have [at least] 2 blackstart stations in Portugal, both were used.
- ncruces 2y ago
  I expect those two (one CCNG, one hydro) are the protocol for a black start, but other hydros should have the capability should those fail.
StopDisinfo910 2y ago
As the press release you linked points out, the black start plan Spain trains on uses nuclear energy supplied from France to re-energize the power plants. "Luckily", France is at an historically high level of production capacity at the moment and the connection between the two countries was reestablished fast. According to RTE (French network manager), the interconnection was maxed yesterday at around 3GW of power. Sadly, while Spain is part of CESA, it's not very well connected. I wouldn't be surprised if one the takeaway from the whole incident is that more interconnections are needed.
prmoustache 2y ago
> It's one reason it's a good idea to always have some cash at home. More than cash it was important yesterday to have the following in case it would have lasted longer: - a battery powered am/fm radio with spare new batteries - some candles and matches - food reserves for a few days that don't need refrigeration: bread, anything in can, pasta, rice... - some kind of gaz or alcohol stove, dry wood or bbq charcoal: you can always make a fire in the middle of the street where there is no risk of burning things around. - water reserve (I always have like 24L of drinking water) and since I hate waste I regularly fill jerrycans when waiting for hot water in the shower that I use for manual washes (kitchenware or gears).
HexPhantom 2y ago
It's wild how much modern life depends on these insanely delicate chains of events going perfectly
bArray 2y ago
Just to add, I was at a University campus when the entire building's electrics went out and there was a significant pull due to relatively powerful computers in every room. They initially tried to bring the building back online altogether and failed. Then they tried to bring it back in sections and failed too. In the end they ended up going into each lab, turning every computer off at the wall to bring each lab's power back, and then turn each computer on one by one. I can only imagine the difficulty of bringing large parts of the grid back online, that rush current must be immense.
- LorenPechtel 2y ago
  Yup. I used to work with a factory that had a bunch of really big machines. Turn everything on at once and the transformer out on the pole self-destructed. Note that the breakers didn't pop--the startup transient was short enough. The power company wasn't happy. Or look at Apollo 13. The astronauts had turned off everything possible because they had lost their generator and only had their batteries. And it took a lot of furious planning by the guys on the ground to come up with a sequence of turning things back on that didn't cause the peak draw to go too high. Can't go too fast or it trips. Can't start too early because the power is limited, but can't start too late because the systems have to be up when they hit the atmosphere.
zestyping 2y ago
I'm confused. Would the start ever have to be truly black? Wouldn't water always be driving hydroelectric turbines, generating some electricity? Solar panels generate electricity without requiring input. I understand that synchronizing AC is not trivial, I'm only questioning the part about whether the start is truly black.

leomca 2y ago

It seems Spain lost 15GW of load, but is still running 10GW of load: https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/show?name=&defaultValue=false&viewType=GRAPH&areaType=BZN&atch=false&dateTime.dateTime=28.04.2025+00:00|CET|DAY&biddingZone.values=CTY|10YES-REE------0!BZN|10YES-REE------0&dateTime.timezone=CET_CEST&dateTime.timezone_input=CET+(UTC+1)+/+CEST+(UTC+2) https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/sh... Would this suggest the grid hasn't snapped apart, or is it just not possible to tell from the data? Coal, pumped hydro, and nuclear generation all went to 0 around the same time, but presumably that's those sources being disconnected from the grid to balance demand? https://transparency.entsoe.eu/generation/r2/actualGenerationPerProductionType/show?name=&defaultValue=false&viewType=TABLE&areaType=BZN&atch=false&datepicker-day-offset-select-dv-date-from_input=D&dateTime.dateTime=28.04.2025+00:00|CET|DAYTIMERANGE&dateTime.endDateTime=28.04.2025+00:00|CET|DAYTIMERANGE&area.values=CTY|10YES-REE------0!BZN|10YES-REE------0&productionType.values=B01&productionType.values=B02&productionType.values=B03&productionType.values=B04&productionType.values=B05&productionType.values=B06&productionType.values=B07&productionType.values=B08&productionType.values=B09&productionType.values=B10&productionType.values=B11&productionType.values=B12&productionType.values=B13&productionType.values=B14&productionType.values=B20&productionType.values=B15&productionType.values=B16&productionType.values=B17&productionType.values=B18&productionType.values=B19&dateTime.timezone=CET_CEST&dateTime.timezone_input=CET+(UTC+1)+/+CEST+(UTC+2) https://transparency.entsoe.eu/generation/r2/actualGeneratio...

mike_hearn 2y ago
They're definitely doing a black start: https://x.com/RedElectricaREE/status/1916818043235164267 https://x.com/RedElectricaREE/status/1916818043235164267 We are beginning to recover power in the north and south of the peninsula, which is key to gradually addressing the electricity supply. This process involves the gradual energization of the transmission grid as the generating units are connected. I see load dropping to zero on that graph, or rather, load data disappears an hour ago. If the grid frequency goes too far out of range then power stations trip automatically, it's not an explicit decision anyone takes and it doesn't balance load, quite the opposite. A station tripping makes the problem worse as the frequency drops even further as the load gets shared between the remaining stations, which is why grids experience cascading failure. The disconnection into islands is a defense mechanism designed to stop equipment being too badly damaged and to isolate the outage.
- leomca 2y ago
  Interesting, but in terms of load I think think the data may just be delayed by ~1 hour. Switching to UTC, to avoid timezone confusion, it's currently 13:10: Last actual load value for Spain at 12:15: https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/show?name=&defaultValue=false&viewType=TABLE&areaType=BZN&atch=false&dateTime.dateTime=28.04.2025+00:00|UTC|DAY&biddingZone.values=CTY|10YES-REE------0!BZN|10YES-REE------0&dateTime.timezone=UTC&dateTime.timezone_input=UTC https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/sh... Last actual load value for France at 12:00: https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/show?name=&defaultValue=false&viewType=TABLE&areaType=BZN&atch=false&dateTime.dateTime=28.04.2025+00:00|UTC|DAY&biddingZone.values=CTY|10YFR-RTE------C!BZN|10YFR-RTE------C&dateTime.timezone=UTC&dateTime.timezone_input=UTC https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/sh...
  - mike_hearn 2y ago
    It may also be a measurement artifact. Looking at the generation by type page, https://transparency.entsoe.eu/generation/r2/actualGenerationPerProductionType/show https://transparency.entsoe.eu/generation/r2/actualGeneratio... Everything dropped to zero except wind and solar, which took huge hits but not to zero. I expect those have been disconnected too, as they cannot transmit to the grid without enough thermal plant capacity being online, but if the measurement at some plants of how much they're generating doesn't take into account whether or not they were disconnected upstream they may still be reporting themselves as generating. You can't easily turn off a solar plant after all, just unplug it. Either that, or they're measuring generation and load that's not on the grid at all.

martinald 2y ago

Interestingly it seems that the black start drill is considering a smaller zone of impact than what has happened here. Also I suspect there is far more renewables on the grid now than in 2016. This is potentially the first real black start of a grid with high renewable (solar/wind) penetration that I am aware of. Black starts with grids like this I imagine are much more technically challenging because you have generation coming on the grid (or not coming on) that you don't expect and you have to hope all the equipment is working correctly on "(semi)-distributed" generation assets which probably don't have the same level of technical oversight that a major gas/coal/nuclear/hydro plant does. I put in another comment about the 2019 outage which was happened because a trip on a 400kV line caused a giant offshore wind farm to trip because its voltage regulator detected a problem it shouldn't have tripped the entire wind output over. Eg: if you are doing a black start and then suddenly a bunch of smallish ~10MW solar farms start producing and feeding back in "automatically", you could then cause another trip because there isn't enough load for that. Same with rooftop solar.

genewitch 2y ago
Grid tied solar won't put power into the grid when the grid is down. It's the one reason I didn't grid tie. Non tied solar won't affect the grid at all. So this is a non-issue. Grid tie requires the grid to tie to, otherwise it can't synchronize. So it stays disconnected.
- martinald 2y ago
  Youre missing my point. I know that; I mean if you are restarting the grid and say you have a segment you think has 5MW of load that will come online, but you connect it and a minute or two later suddenly 2MW of grid tied solar detects the grid and starts exporting and you now have 3MW of load it is going to make it much more tricky to balance the restart. I'm not sure how much of a problem this is in reality but it seems to me restarting a grid is made much more tricky when you have millions of generation assets with no control over.
  - christkv 2y ago

qyckudnefDi5 2y ago

This guy grids!

chippiewill 2y ago

The frequency aspect of a black start is presumably a bit easier in Europe because there's an interconnected synchronous grid so they can bootstrap it from France essentially. It's far more problematic for the UK because all the interconnects are DC.

padjo 2y ago
I was recently told by an electrical engineering lecturer that the black start plan here in Ireland is to use the DC interconnectors with the UK to provide startup power to a synchronous generator.
- ajmurmann 2y ago
  Did he mention the UK's plan in case they have a full blackout?
  - pjc50 2y ago
    https://www.nationalgrid.com/sites/default/files/documents/High%20Level%20Black%20Start%20Strategy.pdf https://www.nationalgrid.com/sites/default/files/documents/H... (2017, probably current)
    - InsomniacL 2y ago
      "it is prudent that the cost of maintaining Black Start Capability should demonstrate value to the end consumer" That sounds like someone explaining why the solution is so bad, before describing what the solution is.
      - pbhjpbhj 2y ago
        To me it sounds like an energy company attempting to excuse lack of spending on infrastructure whilst paying out millions to C-suite in bonuses and millions more to shareholders whilst arguing prices have to rise because they can't afford to spend on infrastructure...

celticninja 2y ago

cash is all well and good but if the tills dont work most stores wont be able to serve you. corner shops, vape shops and barbers will be ok though

genewitch 2y ago
Stores with tills and freezers etc will have power for the tills but the backing network for payments probably won't be up. That's the concern with being cashless. They can accept cashxl, but no one has any.
petertodd 2y ago
The tills have keys to manually open them, and you can just record transactions on pen-and-paper if needed and enter them later. I've seen plenty of businesses do exactly that during power outages. Not as fast. But totally doable.
torcete 2y ago
I was able to buy some groceries and pay with card. The tills had a battery backup and the network infrastructure that supports card payments was apparently working. That said, lots of people hit the cafés and had to resource to cash payments. There was also lots of people buying bottled water at the shops. So basically, you could divide people in two groups. Those that took it like an extra Sunday, and those that took it like the beginning of a war or something :-D
- diggan 2y ago
  Most people around here seemed to be balanced. Yes, a lot of people outside at bars/cafes enjoying a spare Sunday, but also preparing quietly and slowly in case it takes longer time. All the stores ran out of batteries, radios and such, at least the ones we visited, and I only managed to make one card payment around 13:30 sometime, after that internet stopped working 100% until the night. Almost no super markets were open around here, maybe one or two who had generators it seems.

floatrock 2y ago

Practical Engineering did a really great video a few years ago on why black starts are hard, complete with a tabletop demo about the physics of synchronizing large spinning generators: https://www.youtube.com/watch?v=uOSnQM1Zu4w https://www.youtube.com/watch?v=uOSnQM1Zu4w

apexalpha 2y ago

Would this be relevant here? Spain is connected to the French grid, and probably also Morocco. The entire EU runs on one synchronised grid so from that perspective a single 'province' went offline, not the grid.

crmd 2y ago
It would be essentially the same thing as a grid black start, except that the first breaker to close has the European grid on its primary side, instead of a freshly started generator under your control. The complex process of configuring the transmission network to bring grid power to each power plant in succession is the same.
lysace 2y ago
The continental part of the EU runs on one synchronized grid. The Nordics (except half of Denmark and, uh, Iceland, Greenland etc) runs on a separate synchronized grid. https://en.wikipedia.org/wiki/Continental_Europe_Synchronous_Area#/media/File:European_electricity_grid.svg https://en.wikipedia.org/wiki/Continental_Europe_Synchronous...
jameshart 2y ago
Re: probably also Morocco: yes, there is an interconnect across the Strait of Gibraltar https://en.wikipedia.org/wiki/Spain-Morocco_interconnection https://en.wikipedia.org/wiki/Spain-Morocco_interconnection
- dom96 2y ago
  Speaking of Gibraltar, it's interesting that its electricity grid isn't connected to Spain/EU's grid. So they are completely unaffected.

mlsu 2y ago

Does solar power make this process easier or harder? I know that with thermal plants you have a spinning mass that you have to synchronize, and phase shift is used to assess how hard the plant is working (and whether to trip a disconnect as we see here) But with solar, how is the synchronization provided? In like a giant buck? Or in software somehow? Does the phase shift matter as much as in the electromechanical systems? My intuition is that solar would make the grid harder to keep stable (smaller mass spinning in sync) but also may offer more knobs to control things (big DC source that you can toggle on/off instantly.. as long as sun is out). But I don’t actually know.

mike_hearn 2y ago
Most solar and wind plants follow the inertial lead of the thermal plants. They can't synchronize without enough thermal generation being online. Supposedly there are efforts to change that, I don't know enough about grid engineering to say how far along that might be in Spain.
pjc50 2y ago
Mike_hearn's comment was grey but was correct: phase following is indeed done through software in the inverter. Phase matching is still required, wherever the phase difference is not zero there is a deadweight loss of power as heat. Currently the main driver of battery deployments is not so much energy price time arbitrage as "fast frequency fresponse": you can get paid for providing battery stabilization to the grid. (for the UK not Spain: https://www.axle.energy/blog/frequency https://www.axle.energy/blog/frequency )
- mlsu 2y ago
  Oh that is fascinating. So if you have a smarter solar panel, or a smart battery, you can stabilize the grid. I’m assuming that all of the traditional software complexity things in distributed systems apply here: you want something a little bit smart, to gain efficiency benefits, but not too smart, to gain robustness benefits. My intuition is that bringing the market into it at small timescales probably greatly increases the efficiency significantly but at the cost of robustness (California learned this “the hard way” with Enron) > Phase matching is still required, wherever the phase difference is not zero there is a deadweight loss of power as heat If the electronic controller is “ahead of” (leading) the grid, then that heat would come from the solar plant; if it is “behind” (following) then that heat comes from the grid. Is that right? And likely, solar plants opted for the simplest thing, which is to always follow, that way they never need to worry about managing the heat or stability or any of it. I wonder if the simplest thing would be for large solar plants to just have a gigantic flywheel on site that could be brought up via diesel generators at night…

bob1029 2y ago

> Thermal plants require energy to start up It's not just about the power. System components cannot be brought to operating temperatures, speeds and pressures faster than mechanical tolerances allow. If a thermal plant is cold & dark, it can take days to ramp it to full production.

pfdietz 2y ago
That's true of some kinds of thermal generators, but not all. Simple cycle gas turbines can come up very quickly (think jet engines). Or your car's engine.
- bob1029 2y ago
  > Simple cycle gas turbines can come up very quickly In some cases yes. Modern combined cycle plants can take as little as 30 minutes to ramp to full output. Older designs can take upward of 4-6 hours. If you have steam as an indirection, that's when things take a really long time. Natural gas turbines are a more direct cycle.
  - pfdietz 2y ago
    Combined cycle turbines do have a steam component (that's where the word "combined" comes from). The waste heat from the combustion turbine front end is used to make steam in the back end.
    - bob1029 2y ago
      Combined cycle turbines are not primarily powered by steam. It is a secondary consequence of their operation. This contrasts with nuclear and coal plants where steam is the prime mover.
      - genewitch 2y ago
        That's what they said, though
      - 2y ago

Vox_Leone 2y ago

The fewer resources we dedicate to grid resilience and modernization, the harder black starts become. And as grids get more complex and interdependent, recovering from total failure becomes exponentially harder. A rare but sobering opportunity to reflect on something we usually take for granted: electricity. We live in societies where everything depends on the grid — from logistics and healthcare to communications and financial systems. And yet, public awareness of the infrastructure behind it is shockingly low. We tend to notice the power grid only when it breaks. We’ve neglected it for decades. In many regions, burying power lines is dismissed as “too expensive.” But compare that cost to the consequences of grid collapse in extreme weather, cyberattacks, or even solar storms — the stakes are existential. High-impact, low-frequency events are easy to ignore until they’re not.

roughly 2y ago
Yeah, this is the turkey’s dilemma - life on a farm is a lot better than life in the wild for 51 out of the 52 weeks of the year. Most of our modern economy and systems are built to reduce redundancy and buffers - ever since the era of “just in time” manufacturing, we’ve done our best to strip out any “fat” from our systems to reduce costs. Consequently, any time we face anything but the most idealized conditions, the whole system collapses. The problem is that, culturally, we’re extremely short-termist- normally I’d take this occasion to dunk on MBAs, and they deserve it, but broadly as a people we’re bad at recognizing just how far down the road you need to kick a can so you’re not the one who has to deal with it next time and we’ve gotten pretty lazy about actually doing the work required to build something durable.
- Slava_Propanei 2y ago
  [dead]
- maxerickson 2y ago
  What are some examples of modern system collapse? We've had substantial disruptions, but they've not been particularly irrecoverable or sustained.

dv_dt 2y ago

I would think that renewable infrastructure could be the fix, at least if you start installing larger battery capacity to meet renewable store and usage shifts, the grid essentially is installing the resources that can also be used to respond to & contain sudden source losses and prevent cascades.

oilman 2y ago

In another life I worked as an engineer commissioning oil rigs and I’ve seen how tricky even a small-scale black start can be. On a rig, we simulate total power loss and have to hand-crank a tiny air compressor just to start a small emergency generator, which then powers the compressors needed to fire up the big ~7MW main generators. It's a delicate chain reaction — and that's just for one isolated platform. A full grid black start is orders of magnitude more complex. You’re not just reviving one machine — you’re trying to bring back entire islands of infrastructure, synchronize them perfectly, and pray nothing trips out along the way. Watching a rig wake up is impressive. Restarting a whole country’s grid is heroic.

zdragnar 2y ago
I can appreciate the ability to revert to hand cranking an air compressor, yet I can't help but feel that the 99.99% of events, you'd be better served with keeping a two stroke gas engine ready to go. Air compressors tend to have parts just as or more vulnerable to environmental factors, and you get a lot more power for less elbow grease out of a two stroke.
- Symbiote 2y ago
  A power station can start a decently large generator with batteries. Maybe there are other concerns for an oil rig.
  - tonyarkles 2y ago
    Batteries are great when they have charge. What happens if the generator doesn't want to start the first, second, and third time? How many start attempts do you get before the batteries are dead? The hand-pumped air compressor is the tool of last resort. You can try an engine start if there's someone there who's able to pump it. You don't have to worry about how much charge is left in your batteries or whether or not the gasoline for the 2-stroke pump engine has gone stale. It's the tool that you use as an alternative to "well, the batteries are dead too, guess we're not going to start the engine tonight... let's call the helicopters and abandon ship"

egorfine 2y ago

Ukraine can share some experience with forced black starts.

petertodd 2y ago
Ukraine is interesting in this context because there are so many generators. In the richer parts of Odesa I've even found it hard sometimes to tell whether or not the grid power is on as literally every single building had sufficient backup generators to keep the lights on (also, many big businesses seemed to run their generators even when the power was on, I presume as a civic-minded way to add generation capacity overall and avoid interruptions when the power flips on and off).
- egorfine 2y ago
  Generators are not in sync with the grid though. And that's the hardest part. Here's why generators were running here despite the grid being available. A generator has a very short lifetime and in order to prolong it, some owners learned to run those in the very optimal schedule. Which sometimes requires a minimum amount of time to run in a single cycle. Thus if you started it you are committed to run for X hours.
  - petertodd 2y ago
    > Generators are not in sync with the grid though. And that's the hardest part. I know. What's interesting re: Ukraine is because there are so many generators there are more options to getting power sufficiently restored for normal life than just rushing to restore the entire grid. > Thus if you started it you are committed to run for X hours. I'm skeptical this is the main reason. While fast starting a diesel generator is hard on it, there are other, slower, ways to start big diesel generators with minimal impact on lifespan. The blackouts in Ukraine are almost all on a schedule, so big buildings with dedicated staff and expensive generators can and do startup their generators in advance (I've personally heard this happen on occasion - big generators spooling up multiple minutes in advance of the scheduled blackout). Also, it makes sense to turn on generators in advance anyway: gives you a chance to diagnose any issues.

gwbas1c 2y ago

Spain is part of the largest interconnected grid in the world: https://en.wikipedia.org/wiki/Continental_Europe_Synchronous_Area#/media/File:European_electricity_grid.svg https://en.wikipedia.org/wiki/Continental_Europe_Synchronous... Does this really qualify as "black start" when they can rely on the bigger EU grid?

throwup238 2y ago
Yes because they have to bring it all back up in phases so that they only face the load spike* from one interconnect at a time, which can take some time and can fail if there’s unknown damage like the GP said. It really depends on the region though because almost all large hydroelectric dams are designed to be primary black-start sources to restore interconnects and get other power plants back up quickly in phase with the dam. i.e. in the US 40% of the country has them so it’s relatively easy to do. The hardest part is usually the messy human coordination bit because none of this stuff is automated (or possible even automatable). * the load spike from everyone’s motors and compressors booting up at the same time
- pbhjpbhj 2y ago
  Presumably emergency phone comms still work(?) so they could issue instructions to do a phased (heh!) restart to avoid every fridge/air conditioner/whatever restarting at once. Not sure how successful that would be however.
  - throwup238 2y ago
    There’s usually two parallel processes going on during a black-start: spinning the power plants back up to get them synchronized to the grid, and getting power back out to all the consumers. Power plants have breakers that disconnect them from consumers which they keep disconnected until independent connections to other power plants allow them to spin up their massive turbines and synchronize them to the grid. There’s also tons of other downstream breakers at substations which will be in various states of functionality. The power plants with direct connections have hard lines and black-start procedures that get power out to the most important customers like telecom infrastructure, which provides the rest of the comms. In a real world full restart it’s going to mean organizing workers at many substations to babysit old infrastructure so cellular is pretty much mandatory.

larusso 2y ago

I actually have no real clue how and where Spain/Portugal is connected to rest of Europe but could they also restart North to south with help from the grid from France?

kennysmoothx 2y ago

This is literally like Factorio. If your Factory uses too much power, theres not enough energy to run the power plants generation, which decreases your power production. Death spiraling until theres no power. You have to disconnect the factory, and independently power your power plants back up until you have enough energy production to connect your factory up again.

eddd-ddde 2y ago
In my server I hooked up a sound alarm to a set of capacitors. Too low of a charge indicates higher power consumption than production, allowing you to unplug certain low priority loads. I also have some emergency coal generators ready to go at the flick of a switch if needed. Man I need to go play some more.
baq 2y ago
They should allow you to build automatic rate-of-change-of-power-generation breakers...
- abdullahkhalids 2y ago
  Factorio has logic circuits which allow you to do this.
shagie 2y ago
Capacitor circuit network warning for alarm for "main grid is drawing on this bank - bad things may happen if capacity is not increased." Another "trick" is those burner inserters are black start capable. They can pick up fuel and feed themselves to keep running without an electrical network. I also tend to put Schmitt triggers in low priority areas. They've got a battery on the main grid next to them and if the battery drops below 50% power they remain off until it goes back above 75% power.
- tux3 2y ago
  Can't wait for a Factorio mod that simulates the grid frequency and damages equipment when it drifts too far!

belter 2y ago

One of the Administrators of the REN, the Portuguese electric supplier is currently giving a press conference. Confirmed they are in scenario of restart from black start. - Cause of event not known yet. - They noticed power oscillations from the Spanish grid that tripped safety mechanisms in the Portuguese grid. At the time, due to the cheaper prices, the Portuguese grid was in a state of importing electricity from Spain. - They are bringing up multiple power systems and the Portuguese grid is able to supply 100% of needs if required. It was not configured in such a state at the moment of event. - They had to restart the black start more than once, since while starting, noticed instabilities in some sectors that forced them to restart the process. - Time for full recovery unknown at this time, but it will take at least 24 hours.

lastdong 2y ago

Can’t upvote enough! Thank you!

beefnugs 2y ago

And if you are a mere mortal in this world, play Factorio : Space age expansion on the planet Aquilo. To learn the precious importance of reliable multi stage power bootstrapping

anovikov 2y ago

Power never went out in a country completely. At the lowest point consumption was ~40% of normal for that time of day. Ukraine went through many black starts in the first winter of Russian strikes against energy. I guess they built a skill of recovering it quickly enough that it started happening faster and easier every next time.

Fire-Dragon-DoL 2y ago

Oh damn, I thought this was a problem only in Dyson Sphere Program, lol

sgt101 2y ago

I did a project where we predicted transformer failure (they blow up!) from changes in the oil that they have in them (its insulative properties suddenly change). This was 30 years ago so it's all a bit fuzzy, but the one thing that really stuck with me was the story that the SME that we were working with had about the UK grid teetering on the edge of failure during the "great storm" of 1987. His telling was that they really were unsure if they would get it back at all!

bsder 2y ago

>It's one reason it's a good idea to always have some cash at home. Most places are so dependent upon electricity that they can't even take cash during a blackout. And they don't even have the mechanical machines to take a credit card imprint anymore.

brewdad 2y ago
The last of my raised number credit cards went away last month. Those old machines will only get a blank rectangle from me. Sad, because I did actually use one of those about 3 years ago when a rural gas station had power but no network.
- folmar 2y ago
  Network is not required for mag-strip or emv operations, but the terminal has to be configured accordingly (and has to have some way to send batch of transactions for settling). It's less common now, but fully supported.
  - TrueSlacker0 2y ago
    If the network comes back up within 24hrs.

isolli 2y ago

What does synchronization mean in this context, and why is it needed?

seabrookmx 2y ago
The bulk of the power grid is alternating current (AC), and the frequency of the resulting sine wave needs to be synchronized with the other parts of the grid it is connected to.
Tempest1981 2y ago
The grid uses AC (not DC), running at 50 Hz (cycles per second). So the voltage is going up and down at that frequency, in a sine-wave pattern. If you try to connect another generator to the grid, it needs to be at the same point (phase) in the sine-wave cycle, so that its power contribution is added, not subtracted. If it's not in sync, huge currents can flow, causing damage. Sort of like connecting jumper cables backwards.

hnthrowaway0315 2y ago

I wonder if someone could build a realistic scenario into a game -- let's say some sort of smaller scale black start for, say, a space station. And throw in a unknown computer architecture for the in-game computers so that players need to RTFM to figure out how it works.

Xorakios 2y ago
Awesome idea!
sterlind 2y ago
make it so! the fun of reverse engineering systems I don't understand to fix them is half the reason I joined this industry.
dxuh 2y ago
I tried to make something like this (not power specifically) for a game jam once: https://pfirsich.itch.io/arbitrary-complexity https://pfirsich.itch.io/arbitrary-complexity I took down the servers though, so you probably can't easily try it. I don't know if I added a way to configure the lobby server. I should have! It's open source though. And there is a video about that thing on my YouTube: https://www.youtube.com/watch?v=6TPgfa7LbiI https://www.youtube.com/watch?v=6TPgfa7LbiI The game is bad and nothing of what we planned on doing actually made it into the game. The video is long and boring too. But maybe someone finds this cool and is inspired by this and makes a game like this. The first 15 minutes of the game were actually about getting the ship moving, first by reading the manuals of half a dozen different ship systems and then following some procedure outlined in those manuals (parts of which were simply incorrect), maybe having to do some things in sync with your other players and stuff like that. I think it would have been cool to add multiple reactors and start them up in sync and stuff. The different ship systems were actually Lua programs that interacted via a message bus. So kind of a unknown computer architecture?
- hnthrowaway0315 2y ago
  Wow this really looks interesting.

bill3478 2y ago

A true black start has several factors (which make it difficult and notable): 1. The grid has to fully collapse with no possibility of being rescued by interconnection 2. As a result, a generation asset has to be started without external power or a grid frequency to synch to 3. An asset capable of this is usually a small one connected to a lower voltage network that has to then backfeed the higher voltage one 4. Due to the difficulty of balancing supply/demand during the process, the frequency can fluctuate violently with a high risk of tripping the system offline again None of this applies in yesterday's case: The rest of the European synchronous grid is working just fine. News reports stated Spain restored power by reconnecting to France and Morocco. By reestablishing the HV network first, they can directly restart the largest generation asset with normal procedures. As they bring more and more load or generation online, there's little risk of big frequency fluctuations because the wider grid can absorb that.

fredo2025 2y ago

It’s not just restarting it that’s difficult. There are the side effects like what happened in NY: https://en.wikipedia.org/wiki/New_York_City_blackout_of_1977 https://en.wikipedia.org/wiki/New_York_City_blackout_of_1977 > have some cash at home For maybe the first 24 hours at a grocery store, and then not so sure. Would your neighbors sell you supplies and food? Maybe not? And so many places now depend on cashless transactions and doubtful they have pen, paper, lockbox, and safe as a contingency plan.

greggsy 2y ago

Sounds like my last Satisfactory build.

bigbacaloa 2y ago

[dead]

fifteen1506 2y ago

Black Start achieved! Not at 100% yet. Apparently we have [at least] 2 blackstart stations in Portugal, both were used.

ncruces 2y ago
I expect those two (one CCNG, one hydro) are the protocol for a black start, but other hydros should have the capability should those fail.

StopDisinfo910 2y ago

As the press release you linked points out, the black start plan Spain trains on uses nuclear energy supplied from France to re-energize the power plants. "Luckily", France is at an historically high level of production capacity at the moment and the connection between the two countries was reestablished fast. According to RTE (French network manager), the interconnection was maxed yesterday at around 3GW of power. Sadly, while Spain is part of CESA, it's not very well connected. I wouldn't be surprised if one the takeaway from the whole incident is that more interconnections are needed.

prmoustache 2y ago

> It's one reason it's a good idea to always have some cash at home. More than cash it was important yesterday to have the following in case it would have lasted longer: - a battery powered am/fm radio with spare new batteries - some candles and matches - food reserves for a few days that don't need refrigeration: bread, anything in can, pasta, rice... - some kind of gaz or alcohol stove, dry wood or bbq charcoal: you can always make a fire in the middle of the street where there is no risk of burning things around. - water reserve (I always have like 24L of drinking water) and since I hate waste I regularly fill jerrycans when waiting for hot water in the shower that I use for manual washes (kitchenware or gears).

HexPhantom 2y ago

It's wild how much modern life depends on these insanely delicate chains of events going perfectly

bArray 2y ago

Just to add, I was at a University campus when the entire building's electrics went out and there was a significant pull due to relatively powerful computers in every room. They initially tried to bring the building back online altogether and failed. Then they tried to bring it back in sections and failed too. In the end they ended up going into each lab, turning every computer off at the wall to bring each lab's power back, and then turn each computer on one by one. I can only imagine the difficulty of bringing large parts of the grid back online, that rush current must be immense.

LorenPechtel 2y ago
Yup. I used to work with a factory that had a bunch of really big machines. Turn everything on at once and the transformer out on the pole self-destructed. Note that the breakers didn't pop--the startup transient was short enough. The power company wasn't happy. Or look at Apollo 13. The astronauts had turned off everything possible because they had lost their generator and only had their batteries. And it took a lot of furious planning by the guys on the ground to come up with a sequence of turning things back on that didn't cause the peak draw to go too high. Can't go too fast or it trips. Can't start too early because the power is limited, but can't start too late because the systems have to be up when they hit the atmosphere.

zestyping 2y ago

I'm confused. Would the start ever have to be truly black? Wouldn't water always be driving hydroelectric turbines, generating some electricity? Solar panels generate electricity without requiring input. I understand that synchronizing AC is not trivial, I'm only questioning the part about whether the start is truly black.