27 ms·
Texas grid flags risks as data centers, crypto sites fail voltage tests
- jeffbee 5mo agoThis sounds like a good reason to have a lot of batteries.
- weird-eye-issue 5mo agoA lot would probably be an understatement. Aren't most batteries in data centers designed to just hold load for seconds or a few minutes at a time while generators start up?
- zephen 5mo agoYeah, different batteries, different purposes. Keeping the data center up is completely different from keeping the grid up. Not only are the batteries too small; they're also on the wrong side of the disconnect.
- jeffbee 5mo agoIf the problem is an instant disconnect of a large load, switching that large load into charging batteries at that instant, for a minute or so, feels like a solution.
- weird-eye-issue 5mo agoWhat?
- jeffbee 5mo agoWhy is this confusing? Batteries are both generator and load. If the grid needs load to stabilize frequency and voltage, batteries can become loads instantly.
- weird-eye-issue 5mo agoOh, okay. I was confused because I was talking about batteries on the side of the data center.
- forgetfreeman 5mo agoI'd rather see legislation banning crypto mining and AI data centers from the public grid entirely. No sense in forcing the broader public to subsidize them.
- MBCook 5mo agoThe problem with that is one of the best things we have to control pollution at power plants is the rules that go into place when connecting to the US grid (I know TX is different). I really don’t want to incentivize private power plants that aren’t on grid. Or just running tons of industrial sized generators instead. If we’re going to allow enough of this stuff to be built that it can destabilize things why not require they behave and don’t stop off like that? Some sort of organized draw down? And if they don’t? Mandatory cutoff for X amount of time. Weeks/months.
- toomuchtodo 5mo agoBan private fossil generators above a certain size without a license. You can just do things. They can build as much solar and batteries as they want. https://www.datacenterdynamics.com/en/news/google-confirms-1gw-data-center-campus-near-detroit-michigan-partners-with-dte-energy-on-27gw-power-generation/ https://www.datacenterdynamics.com/en/news/google-confirms-1... https://www.datacenterdynamics.com/en/news/google-announces-first-data-center-in-minnesota/ https://www.datacenterdynamics.com/en/news/google-announces-...
- dpark 5mo agoSpaceX is running a bunch of “portable”, high pollution gas generators in Memphis, TN specifically to get around the regulation you’re describing. Elon definitely got the “you can just do things” memo.
- toomuchtodo 5mo agoRight, that’s what I was thinking of when I wrote my comment. Regulate back. If there is no will to do so, well, that’s a choice. Write the law, pass the law, aggressively enforce civil and criminal penalties for violations. They haul gas generators in without a license? Confiscate and tear them down for scrap (which will be painful, as these turbines are in short supply and their manufacturers are backlogged years into the future), in accordance with law you pass. Hold the utility liable if they provide a fossil gas pipeline connection. Humans like Elon may not care, but utilities have something to lose. Find “one throat to choke” as the saying goes. It is not politically easy, but it is logistically straightforward. https://en.wiktionary.org/wiki/one_throat_to_choke https://en.wiktionary.org/wiki/one_throat_to_choke
- threwrfaway 5mo agoMore scarce resources used to feed the surveillance state's fancy chat bots
- drak0n1c 5mo agoTexas was the leading state in new grid battery installs for the last few years.
- rickydroll 5mo agoI think it's time to put data centers on a power budget. If they want to make more money, they need to become more efficient and eliminate AI fraud, waste, and abuse.
- onionisafruit 5mo agoI’m curious how this works for other large consumers. Do they have some kind of artificial load that lets them gradually reduce consumption instead of doing it all at once?
- threwrfaway 5mo agoIf you're large enough your connection to the grid is a negotiation with an engineering team. The utility will force you to put equipment to correct for power factor (massive capacitor bank), resistive load, etc. The utility also charges commercial users for apparent power (includes reactive power, or that sloshes around setting up a steady state), as opposed to just real power charged to residential users. EDIT: in case your wondering, yes resistor loads is just glorified bunch of short circuits and a fan.
- onionisafruit 5mo agoThat’s interesting. For the rest of us I guess it’s just the law if large numbers that our air conditioners don’t cycle off at the same time
- threwrfaway 5mo agoPretty much. Except the law of large numbers breaks down in correlated events. Like a black out. What really hurts is are all the rotating machines, especially the one-phase ones (fridge, AC, blower). Those have transients that last several cycles and are electrical shorts until the back emf is setup. Utilities will try to roll the power back on in sections to avoid instability, starting with hospitals and those near hydro plants. But you can help put by turning off your stuff during the blackout and when the light come back on walking around the house to turn them back on.
- tikkabhuna 5mo agoIn the UK we have that scenario with people turning on kettles during breaks in TV programs. https://en.wikipedia.org/wiki/TV_pickup https://en.wikipedia.org/wiki/TV_pickup
- m-hodges 5mo ago[dead]
- stldev 5mo agoDo we really need to keep slamming the grid and killing the planet for pseudonymous casino chips? Can someone please merge crypto with llm training/inference somehow?
- threwrfaway 5mo agoHow about an across the board $1/W hook up fee for new customers? Thats about the price of installed capacity per watt. New house with 200 A panel an assumed 30% utilization rate? $3600. New data center with 80% utilization rates at 100MW? $80 million dollars. New 10 GW data center? That'll be $8 billion. It's outrageous that I'm paying an extra fee to export energy to a neighboring state to power a datacenter.
- mindslight 5mo agoYour math on the house seems off by a factor of 4? But 30% utilization seems high, as that would be a $2k/mo electric bill at 20 cents/kWh. Reframing what you're saying, that would be a connection fee that worked out to just under 12 cents per kWh used for the first year, which seems both somewhat reasonable but also probably not going to move the needle much on the large deployments?
- threwrfaway 5mo agoI guessed what the utilization factor for a house was, purposely going for a very high number to show that it wouldn't add too much to the house. Note, In proposing a one time fee, on the capacity to generate power, not the energy.
- mindslight 5mo agoYes, but using the 30% utilization factor I get 200A * 240V * 30% = 14.4kW , meaning a $14400 connection fee rather than $3600 ? $3600 would be a 7.5% utilization factor. And yes, I got the one time fee aspect. My point is that if you're basing this off of expected utilization (rather than say the size of the service regardless of how much energy is used), and that expected utilization is roughly correct, then that one time fee can be thought of as amortized cost per unit energy over a specified amount of time. And that cost isn't even that high if amortized over a year.
- HWR_14 5mo agoTexas has an independent energy grid, so are not exporting energy to them.
- ilaksh 5mo agoThis almost seems like a straw man to me. Isn't the much larger problem the actual increased energy usage and making sure that all of this massive extra cost doesn't just get dumped on consumers? I am a huge proponent of AI actually, but very suspicious that "financiers" are suddenly creating what amounts to an energy tax by finding legal ways to sneak extra fees or rates into our electricity bills to cover build out and commercial usage costs. But as far as smoothing out demand, my (admittedly a layperson) theory is that we need to force them to adapt more solar and wind and at the same time more facilities for handling the variable production from that. Such as more large batteries and a shift to large scale long term storage of renewable fuel like hydrogen or other fuels produced directly from renewable sources. If you have a large production and storage of renewable fuel, then maybe you can build that in such a way that it can handle significant input variations that could include excess grid power.
- jeffbee 5mo agoTexas is the standing proof that "making sure that all of this massive extra cost doesn't just get dumped on consumers" is not a problem. Texas has the most load, the most marginal load added every year, and the cheapest retail power. Load grew 15% in the 5 years to 2024, and retail prices fell by 1¢.
- doctorwho42 5mo agoDo t they also have like the most solar?
- jeffbee 5mo agoMaybe in absolute terms? I haven't checked in a while. In terms of fraction of energy from solar Texas is still far behind California.
- ZeroGravitas 5mo agoThey have more utility solar than California as of last year in absolute terms, they're second if you include distributed behind the meter solar which California has more of. Texas also lead in absolute amounts of wind, gas and coal. They have a lot of generation.
- threwrfaway 5mo agoFor those who dont know why this is important: The loads are slowing down the generators that are burning a well metered amount of fuel to stay at 60Hz. This is a delicate balance since the phase angle must also be spot on. If a generator and the local line disagree on f, phase or V, you have a short circuit. If you lose a large amount of load, your generator will spin up with the excess fuel until the control system re-establishes the right amount of fuel. But now your generators are out of sync! No worry, for small disturbances the dissipative losses sync everything up like syncros on a manual transmission. But the disturance cant be too big! Rotating machines are big and heavy, so the first line of defense is their inertia. But this is a finite (and precious) resource. Contrary to belief, renewables, or generally speaking DC, makes things this stability problem worse. They generate large amounts of power while providing no inertia. You'd think it isn't a big deal since the DC-AC converter can just synthesize whatever is needed. Heck just keep it rigid at 60 Hz with no phase change. Well the later doesn't work - the rest of the grid is no longer at that phase and frequency so you got yourself a short. Furthermore, the DC-AC converter, despite their manufacturers' promise, has no good way to establish what f and phase it should be at during a disturbance (and these magic codes are closed source, believe it or not) Anywho, a large enough loss of load causes the grid to enters into unstable oscillations, causing protective relays to trip causing a zipper effect where the grid goes down. Now restart will take a few days depending on the energy mix (fastest for hydro heavy) Long story short - this is not a trivial problem, and the data-centers can't be allowed to just dump load willy nilly. EDIT: made it clear that the grid killing disturbance is not caused by renewables; not exclusively anyway. Everyone has to play nice or the grid goes down.
- TOMDM 5mo agoWould demanding that large spikey users of energy like data centers implement some sort of demand ramping/isolation from the grid in the form of a massive capacitor bank or flywheel generator between them and the grid help reduce the risk here?
- manarth 5mo agoThe data centres aren't inherently spikey, in general use their consumption is reasonably predictable. However, if a DC detects that the _grid_ is wobbly (voltage or frequency deviations) the DC will disconnect without warning, and switch to its batteries and generators. The grid complains because it's suddenly lost hundreds of MW of load. For the DC to have isolation capability, it would need a load-sink which can consume roughly the same power as the DC in normal operation, and can take in that load at a moment's notice. It's a hard problem to solve, and probably better managed at grid-level than DC-level.
- Pxtl 5mo agoI look forward to hearing from the usual suspects how the inevitable failure is the fault of wind and solar power.
- Rapzid 5mo agoThis all seems "routine". Also the little sound bite about peak demand, Texas has enormous capacity during the summer as well; far, far exceeding demand. It makes total sense to be testing and connecting multi-GW consumers during these months.
- brycewray 5mo ago> Also the little sound bite about peak demand, Texas has enormous capacity during the summer as well; far, far exceeding demand. It makes total sense to be testing and connecting multi-GW consumers during these months. Texan here. Actually, it depends on how hot our summer weather gets and, thus, how much A/C use is in play. When we get into a not-unusual run of multiple 100-degree (F.) days in a row, the available capacity often drops to the point where ERCOT begins issuing alerts about things like suggested times of day when running certain appliances might not be wise. Having lived through the nightmare of the February '21 winter storm outages, I keep the ERCOT dashboard[0] as a bookmark and check it at least once, every day of the year. [0]: https://www.ercot.com/gridmktinfo/dashboards https://www.ercot.com/gridmktinfo/dashboards
- gblargg 5mo agoCharge them the cost of installing mitigations that can pick up the load if they drop it suddenly.
- bob1029 5mo agoI'm a bit confused about why this is a serious engineering problem. If a gigawatt class DC suddenly needs to take its sensitive IT loads off grid, it could be designed with load banks on site to stand in for the IT load. These would be exceptional use only, so the specific cooling technology (obviously we want to boil water) is not much of an ecological concern. A gigawatt will vaporize ~100 gallons of water per second. How long until the grid can adapt? Five minutes? That's not exactly a heroic amount of water for these projects.
- burnt-resistor 5mo ago> could be designed with load banks on site to stand in for the IT load Yes, but that's not what's happening. These giant loads are suddenly going offline and are making it the transmission and generation networks' problem. The DCs aren't going to pay for it when they can socialize the costs and problems onto others. The most obvious answer is these big loads need FFR and M-FFR BESS (battery) capacity near them or on-site to decouple and absorb these huge transient load swings into something the grid can handle. ERCOT should set additional rules that large load customers must use load smoothing rather than simply dumping load at auto-reclosers/auto-cutouts. Not doing so forces the entire grid to be at risk of low inertia transient over-generation.
- ProllyInfamous 5mo ago>why is[/]this a serious engineering problem[?] Because the Texian grid-operator (ERCOT) is predominantly isolated from the Eastern & Western US grids [†], serving mostly-just Texans – this is to avoid US Federal regulations, by being an only-intrastate entity (technically, which is: the worst kind of "correct" [∑]). This is a socially-engineered "problem". This means their needs for maintaining grid inertia (which frequency is a measure-er of) are more difficult, as Texas cannot just purchase any substantial/meaningful electricity – everything must be generated in-house. Worse, the lack of Federal regulations means ERCOT/Texas can create their own rules/regulations/markets for the management of electrical infrastructure. Texas chose to make (IMHO) the most-purely crony-capitalistic market [for buy/sell of MWhours] in worldwide existance – which leads to decannual winterstorm Disasters (totaling hundreds of BBillion$ in damages / hundreds dead). The losses from wasted silicon-ingot depositions (alone) are staggering $,,,.00 – just another cost of doing business within Texas's jurisdiction [ß]. After 2021's storm, some "free market" customers were left with a week of billing costing more than the year's remainder!. Abbott will just blame the windmills and not a cost-incentive fee-structure which does not incentivize cold-weather preventative maintainance. In fact, it does the opposite (since electricity can 25X price-increase). "Tell me the incentive and I'll tell you the outcome." [∑] having grown up and been schooled primarily in Texas, I think the best review I ever agreed with was: "The [L]one Star State" – Texceptionalism will kill us all [†] small DC-DC grid-ties (altogether a low, single-digit percentage of ERCOT's overall capacity) exist with both grids, as well as another smaller with Mexico [ß] which is why most hospitals/datacenters have their own diesel/LNG generators, on-site – in Texas these have even less regulations because: "temporary" see (e.g.): Tesla's multiple Texas facilities' backupsystems – enormous point-source pollutors ---- If you want a good breakdown of the 2021 Texas storm's decimation of ERCOT/Texas/electricity, look for the great logs reddit /u/redditmudder did (in realtime) as this disaster was unfolding. I cannot remember exactly how low the frequency got... but I think we were closer to EU [50hz] than US [60hz] – as grid operators REAPED profits* from their failing infrastructure driving electricity orders-of-magnitude higher. This is all public information; it wasn't "windmills."
- negative_zero 5mo agoIt always seems wild to me how, in the US, something like the grid is just a wild west. "ERCOT said it is reviewing the test failures and drawing up plans to protect the grid from disruptions." Why are they even allowed to connect? / Why are they not kicked off? / Why aren't they being forced to add their own grid inertia?
- onionisafruit 5mo agoThey aren’t being allowed to connect until they get it worked out. At least that’s how I read it.
- klysm 5mo agoERCOT operates under a “connect and manage” interconnection philosophy. This means things can actually get built quickly (unlike other ISOs), but some management is required. Of course, there are some downsides to this approach, but overall I think it’s a superior. Just look at renewable and battery penetration compared to other ISOs.
- Schiendelman 5mo agoThis isn't the US - it's just the state of Texas. The rest of the contiguous United States is on two large grids. Texas has its own. It's up to ERCOT to figure their own mess out. They will. Nobody else really has to care about it.
- tumetab1 5mo agoTaking note of the risk is important for everyone but shouldn't the follow up action be to improve the grid voltage stabilization mechanisms? I have no idea of what entails but this seems more a quality of service issue (provided by the grid) and less about the consumers using the grid.
- josefritzishere 5mo agoI have read in other contexts that Texas has the least stable power grid in America. It has more downtime than even California, with it's annual wildfires. Despite the regulatory appeal, it does argue against data center expansion in Texas generally.
- burnt-resistor 5mo agoThe problem: multi-GW DCs going suddenly offline leading to rapid transient overproduction. What's missing is sufficient FFR and M-FFR BESS (battery) capacity to smooth out these sudden disconnects. There's only about 20 GW of BESS generation capacity total in the system now, and ERCOT needs at least 3x more. Placing large FFR and M-FFR near these giga DCs is what it will take to absorb transient overproduction transients from hurting generators or tripping the grid. This is something DCs should pay for rather than upending how the grid works for problems they cause.