6 ms·
For those who dont know why this is important: The loads are slowing down the generators that are burning a well metered amount of fuel to stay at 60Hz. This i
by threwrfaway 4mo ago
For those who dont know why this is important:
The loads are slowing down the generators that are burning a well metered amount of fuel to stay at 60Hz. This is a delicate balance since the phase angle must also be spot on.
If a generator and the local line disagree on f, phase or V, you have a short circuit.
If you lose a large amount of load, your generator will spin up with the excess fuel until the control system re-establishes the right amount of fuel.
But now your generators are out of sync! No worry, for small disturbances the dissipative losses sync everything up like syncros on a manual transmission.
But the disturance cant be too big!
Rotating machines are big and heavy, so the first line of defense is their inertia. But this is a finite (and precious) resource.
Contrary to belief, renewables, or generally speaking DC, makes things this stability problem worse. They generate large amounts of power while providing no inertia.
You'd think it isn't a big deal since the DC-AC converter can just synthesize whatever is needed. Heck just keep it rigid at 60 Hz with no phase change.
Well the later doesn't work - the rest of the grid is no longer at that phase and frequency so you got yourself a short.
Furthermore, the DC-AC converter, despite their manufacturers' promise, has no good way to establish what f and phase it should be at during a disturbance (and these magic codes are closed source, believe it or not)
Anywho, a large enough loss of load causes the grid to enters into unstable oscillations, causing protective relays to trip causing a zipper effect where the grid goes down.
Now restart will take a few days depending on the energy mix (fastest for hydro heavy)
Long story short - this is not a trivial problem, and the data-centers can't be allowed to just dump load willy nilly.
EDIT: made it clear that the grid killing disturbance is not caused by renewables; not exclusively anyway. Everyone has to play nice or the grid goes down.
- TOMDM 4mo agoWould demanding that large spikey users of energy like data centers implement some sort of demand ramping/isolation from the grid in the form of a massive capacitor bank or flywheel generator between them and the grid help reduce the risk here?
- manarth 4mo agoThe data centres aren't inherently spikey, in general use their consumption is reasonably predictable. However, if a DC detects that the _grid_ is wobbly (voltage or frequency deviations) the DC will disconnect without warning, and switch to its batteries and generators. The grid complains because it's suddenly lost hundreds of MW of load. For the DC to have isolation capability, it would need a load-sink which can consume roughly the same power as the DC in normal operation, and can take in that load at a moment's notice. It's a hard problem to solve, and probably better managed at grid-level than DC-level.
- TOMDM 4mo agoMy intuition is that there would be a fairly stable base load, but doing something like switching on a new training run of a frontier model would be incredibly spiky, thousands of GPUs going from somewhat idle to 100% in seconds.
- cucumber3732842 4mo ago>it would need a load-sink which can consume roughly the same power as the DC in normal operation, and can take in that load at a moment's notice. Liquid rheostat. A big one.
- thelastgallon 4mo ago> The grid complains because it's suddenly lost hundreds of MW of load. For the DC to have isolation capability, it would need a load-sink which can consume roughly the same power as the DC in normal operation, and can take in that load at a moment's notice. Thats why always have all the EVs be connected all the time (except the 20 mins they drive). EVs can provide demand as a service and take excess power whenever its available (instead of solar curtailment), and also provide an immediate source of load when events like this happen. Its a shame US is anti EV, it has the best systems at scale that can be leveraged to transform the entire energy ecosystem.
- WarmWash 4mo agoThis is where a smart grid would really shine, because the utility could drop the electricity price to 0 or even negative and turn on 100,000 air conditioners/heaters to effectively give people free climate control to absorb the load imbalance.
- wyldfire 4mo agoSome of us in Texas are all too familiar with the problem of balancing load with generation, the risk of a cascade failure causing a slow restart. During winter storm Uri, they did a duty cycle where we only had power available for ~6-12 hours at a time on the days it was available. This was apparently to avoid that very problem. So far as I know, the obvious mitigations like winterizing NG generation and/or peering with neighbor grids have not been performed.
- deleted 4mo ago[deleted]
- LadyCailin 4mo ago[flagged]
- namibj 4mo ago10+ MW voltage-source converters that can't do up to around 80% of their nominal capacity as mostly-reactive apparent power with stabilizing synthetic inertia scaled as desired/specified are a mostly software issue, stemming from lack of regulatory pressure incentivizing the engineering complexity of that. Though if you want to do a smoothing action on real power flux you'll have to colocate battery capacity with the converter. Which to be clear is fairly cheap to do as long as you get compensated for the substantial frequency stabilization capacity this represents. I'm talking like 15~120 minutes at converter nominal AC power of battery capacity. The first 10~20% of reactive power are almost free from the converter electronics, btw....
- nutjob2 4mo agoThe software is being written along with the regulations and compensations and it's being rolled out in the real world, just as you described.
- nutjob2 4mo agoGrid scale battery systems are also used to maintain proper synchronization.
- Retric 4mo agoThat’s a simplified and somewhat outdated version, there’s a huge range of technology to mitigate each of these issues but infrastructure generally isn’t free. A turbine does provides inertia for free where you need to pay for a flywheel or battery system. In the end a renewable heavy grid can be extremely resilient due to all the batteries, but smart battery systems need to be incentivize or mandated because ‘dumb’ batteries are cheaper.
- peterus 4mo agoTo add on to this overview, there are a new class of inverters in research called grid forming inverters which don't just follow the rest of the grid. One interesting technique is they can simulate the inertia of a traditional rotating machine/generator. https://iten.ieee-ies.org/journal-featured-article/2025/grid-forming-inverters-a-comparative-study-of-different-control-strategies-in-frequency-and-time-domains/ https://iten.ieee-ies.org/journal-featured-article/2025/grid...
- jillesvangurp 4mo agoGrid forming batteries and inverters are a thing. They can control the frequency on the grid. Just a matter of getting the right equipment. The nice thing with data centers is that they are somewhat flexible. It's not a constant load. Data center operators can choose to reduce load. And if properly engineered, they could do so automatically based on signals from the grid. The issue with outdated grids is that it relies on technology (spinning mass) that's at this point a century old. Which makes it brittle against outages like you describe. The solution is not more spinning mass but batteries and renewables to take the place of that spinning mass. A battery can respond to oscillations in milliseconds. If you then add flexible load that can spin up/down based on the amount of available power, you gain a lot of stability.
- alexey-salmin 4mo ago> The issue with outdated grids is that it relies on technology (spinning mass) that's at this point a century old. Which makes it brittle against outages like you describe. The solution is not more spinning mass but batteries and renewables to take the place of that spinning mass. A battery can respond to oscillations in milliseconds. If you then add flexible load that can spin up/down based on the amount of available power, you gain a lot of stability. Why the spinning mass technology being "century old" (more like "millennia old" but anyways) is a problem somehow? The Newton's First Law didn't change much in the time that has passed. Anyone who tried to "respond to oscillations in milliseconds" knows how hard that problem is because the force you apply is integrated twice before it takes effect. Try stabilizing a swing by pushing it's forward when it's behind the equilibrium point and pushing it backward when it's ahead of it. Now imagine a grid of swings connected by rubber bands and a distributed system of independent actors responding to oscillations. There are much more ways in which this system can diverge rather then converge. Time may prove me wrong, but the arguments like "spinning mass is old therefore should be replaced" certainly won't.
- jillesvangurp 4mo ago> Newton's First Law didn't change much in the time that has passed. No, but technology has moved on quite a bit. Heavy fly wheels are no longer the state of the art here. A few tens of GW of battery capacity (i.e. a few dozen nuclear plants worth of capacity) that can switch on/off in milliseconds can do a lot for grid stability. That's part of the reason why grid operators are rolling out so much batteries. It's not necessarily about supplying energy for a very long time but about smoothing out peaks and dips in energy supply and demand and responding more or less in real time to that. This stuff is basically being rolled out at industrial scale in a lot of places. Australia, China, etc. pretty much run increasingly on mostly renewables. This is no longer as speculative as it would have been ten years ago. Yes, there are engineering challenges with rolling that stuff out in a lot of places. And even more policy and regulation challenges. Actually that is, by far, the #1 challenge in places like the US and Europe. Grid operators are simply structured and incentivized wrong to deal with this stuff efficiently. Texas is actually not doing too bad relative to e.g. California. But they clearly have some challenges still.
- gblargg 4mo agoThe ELI5 version is you're arm-wrestling someone and they suddenly let their arm go limp, so your arm slams down on the table since you can't react that fast.
- flopbob 4mo agoTo build on this analogy, tug of war fits a bit better. Nothing dramatic happens if one person let go, but if half of one team just let loose at the same time without communication, bad things happen.
- ksec 4mo agoThanks. This is a great analogy.
- cogman10 4mo agoI think an even better example is what happened to the Tacoma Bridge. You have an oscillating frequency and phase that needs to be precisely tuned in order for everything to work well. If something is out of sync wrong, you can end up with higher peaks, lower valleys, or flatter locations which can ultimately cause catastrophic failure. In the Tacoma example, the input frequency continued to add onto the bridges motion which ultimately caused it to destroy itself. In an electric grid, a misaligned phase can cause excessive spikes (imagine 480V when you expect 240V) or the generators to ultimately burn themselves out because 2 generators are fighting with each other, one trying to raise the voltage while another is trying to decrease the voltage. The really tricky thing is that load (particularly inductive or capacitive loads) look almost exactly like an unaligned generator. https://www.youtube.com/watch?v=XggxeuFDaDU https://www.youtube.com/watch?v=XggxeuFDaDU
- altairprime 4mo agoI interpreted from that EU grid post mortem that individual generators are coordinated using out of band comms channels that aren’t the power grid itself. Am I mistaken that they do this? Is it done there, but absent here?
- 4mo ago
- hippich 4mo agoWhen you say "short circuit" - is this when wires get hit and make and burn, or this is reference to some other stuff?
- analog31 4mo agoShort circuit is when two sources try to drive the same node in a circuit at different voltages or frequencies. In an idealized circuit (such as where the wires are assumed to have zero resistance), current flow from one source to the other is infinite. In a real circuit, current flow rises to a level where "something has to give." This could include heating the wires between the sources or tripping some kind of protection mechanism. I'm not great with analogies, but imagine a train with two locomotives, and each one is set to run at a different speed. There will probably be a lot of screeching of wheels etc. To make multiple locomotives work on a single train requires engineering them to be synchronized with one another.
- jampekka 4mo ago> Contrary to belief, renewables, or generally speaking DC, makes things this stability problem worse. Is there such belief? My feel is that anybody to whom electrical grid stability has even crossed their minds know this.
- somewhereoutth 4mo agoSo apparently grid forming or synthetic inertia can be provided electronically nowadays, but presumably it would not be too terrible to put a rotating mass in between the (e.g.) solar power source and the rest of the grid? So the electricity from the solar panels runs a motor that turns a generator. I suppose power is lost through friction and other inefficiencies, and you'd need to co-locate the solar panels into a big enough farm.
- chaz6 4mo agoIs it feasible to tie the grid frequency to an external clock reference, such as TAI?
- tenthirtyam 4mo agoI've often thought that the target* grid phase should be encoded into a high frequency signal, say at 50kHz. Generators without inertia can immediately switch to the required phase and inertial systems can work towards it or disconnect their outputs if they stray too far. The problem in my mind is that, as it stands, the signal that everyone has to latch on to is itself affected by load and by different generators latching on with different time constants in a complicated feedback loop. Would having a single authoritative source be an improvement? Would that be a way to eliminate the need for inertia? *I suppose the "target" phase would probably be based primarily on the output from the biggest inertial systems and take into account their ability to adjust phase and frequency.
- bryanlarsen 4mo agoThat's not true. Modern batteries and inverters can be really good at providing synthetic inertia. Proof by example: https://spectrum.ieee.org/grid-scale-battery-scotland https://spectrum.ieee.org/grid-scale-battery-scotland As the article says, this is difficult and adds significantly to the cost. But it is very possible. (It also adds significantly to the cost of spinning mass generators too. Delivering 250% of nominal current in a short circuit situation requires beefing up a lot of components). There are many others.
- namibj 4mo agoActually large data centers at least if done in a vaguely alirack style architecture, can do this with a decent fraction of their nominal power for very little hardware cost, as reactive power and real power add up via Pythagoras (`apparent=sqrt(real^2 + reactive^2)`) to the apparent power (rms voltage times rms current, which is what the 60Hz electronics and 60Hz transformers care about). The first 10-ish % are nearly free. And alirack style datacenters have large 3-phase converters between the grid and some 240 (nowadays often 350) V DC bus, with the battery banks directly (with just fuses and sometimes a little bit of balancing/nudging power (think 10% of battery power rating)) on the bus, and then the servers also directly consuming from that bus. The large converters on the battery bus thus allow synthetically smoothing load transients to the grid using the batteries to smooth that power draw. This has just minor additional wear on the batteries and a small power efficiency impact from hitting through the batteries, both of which are easily paid by anything market-rate of providing that grid service. Because they already need the power electronics and batteries anyways, unlike a utility battery farm that at best can argue day/night load shifting of solar production as the reason for the electronics and batteries to exist. In that same spirit it's also effective to put batteries on the DC bus (between MPPT and inverters) of large solar farms, because they need the electronics anyways and it's actually reducing the required inverter&transformer capacity of the solar farm by peak-shaving.
- digitalsushi 4mo agoCould the GPS provide a correct carrier wave for every electrical system to synchronize with?
- otterley 4mo agoPractical Engineering made an outstanding video on this: "What Really Happened During the Texas Power Grid Outage?" https://www.youtube.com/watch?v=08mwXICY4JM https://www.youtube.com/watch?v=08mwXICY4JM
- mitthrowaway2 4mo agoSynthetic inertia has come a long way. An inverter knows just as well as a rotating generator what frequency and phase it should be generating, because it can mimic exactly the ideal equations of motion of a rotating generator with programmable inertia.