5 ms·

Texas law gives grid operator power to disconnect data centers during crisis

143 points by walterbell 1y ago

Bender 1y ago
As someone that used to help manage data-centers I think this is fine. I had to load test the generators quarterly. It's good to ensure the transfer switches and generators are working and nothing beats a real world load test. The data-centers have contracts with diesel fuel providers to keep the main tanks full. They can run on diesel any time the infrastructure load is high and write off the cost in their taxes check with your tax lawyer. There may even be other tax advantages being the need for generators would be compelled by the state, perhaps a tax lawyer could find ways to make a generator tech refresh get a bigger write-down, write off better noise abatement walls as usage will increase. If a company was running with scissors and did not buy enough generator capacity to run everything then they will have to get their act together now vs. later.
- Shank 1y ago
  This is true, but how long will firm load shed events last, and how many of them will happen? In California, when CAISO has events that lead to firm load shedding, they're predictable, rolling blackouts and everyone knows how long they'll last, and they're assigned on a customer-specific basis. You know your power will be cut, and you know it will be a certain amount of time, and you know roughly where you are in the schedule. I could see operators of datacenters in Texas wondering about this. Also, it's underrated how much critical infrastructure is dependent on datacenters running. Like, are you going to pull someone's EHR system down that serves a local hospital, while keeping the local hospital on a critical circuit?
  - Bender 1y ago
    You know your power will be cut, and you know it will be a certain amount of time, and you know roughly where you are in the schedule. Knowing when the power will be cut will not help unless I am misunderstanding you. If the data-center loses power for even a minute the generators will all fire up and then every ATS will count-down and transfer in 30 seconds. Battery backup only lasts just long enough to do a quick return to service on a failed generator and even that is sketchy at best. A properly engineered data-center can run on reduced generator capacity. Some data-centers are indeed on circuits deemed to be critical but I could see regulations changing this so that they are "business critical" vs. "life support critical" and some changes could be made at substations so that data-centers could participate in shedding. I think you are right that they will be thinking about this and adding to this probably filing preemptive lawsuits to protect their business. Such changes can violate SLA contracts businesses have with power companies and Texas is very pro-business so I can not compare it to California.
    - toast0 1y ago
      > Knowing when the power will be cut will not help unless I am misunderstanding you. If the data-center loses power for even a minute the generators will all fire up and then every ATS will count-down and transfer in 30 seconds. Battery backup only lasts just long enough to do a quick return to service on a failed generator and even that is sketchy at best. A properly engineered data-center can run on reduced generator capacity. If you know when the power will be cut, you can start the generators before the cut, and depending on your equipment, you may be able to synchronize the generator(s) with the grid and switch over without hitting the batteries. I assume big datacenters are on three phase, can you switch over each phase as it crosses zero, or do you need to do them all at once?
      - Bender 1y ago
        At least in the data-centers I helped manage the inverters were in-line running 100% duty-cycle, meaning frequency sync is not required as there is no bypass. The servers never see the raw commercial power. Data-centers in the US are indeed 3-phase. FWIW the big Cats did have controllers that would maintain sync even when commercial power was gone but we did not need it. There wasn't even a way to physically merge commercial and generator power. ATS inputs and outputs were a binary choice. I know what you mean though, the generators I worked with in the military had a manual frequency sync that required slowly turning a dial and watching light bulbs that got brighter with frequency offset. Very old equipment for Mystic Star, post-WWII era equipment. 50's to 90's
        jimmygrapes 1y ago
        Manually syncing several of the MEP012 generators was always far more stressful to me than any physical dangers!
        Bender 1y ago
        I bet. I never messed with the trailer or skid mounted generators. It sounds like you were also USAF. At least modern day noise cancelling headphones are much better. Guessing you probably have tinnitus from working on them. At least I think that is partially where mine came from.
  - vel0city 1y ago
    If they cannot handle the grid power being pulled due to load shedding, they have no business handling critical applications.
    - bluGill 1y ago
      In particular there is no way you can predict when a backhoe will take our your power line. (Even if they call to get lines located - sometimes the mark is in the wrong spot, though most are they didn't call in the first place). There are lots of other ways the power can go down suddenly and nothing you can do about it except have some other redundancy.
  - nradov 1y ago
    All of the major cloud EHRs run in multiple availability zones.
    - mschuster91 1y ago
      The data center might be... but are all fiber routers, amplifiers, PoPs etc. along the line from the datacenter to the hospitals and other healthcare providers backed up as well? Particularly the "last mile" often enough is only equipped with batteries to bridge over small brownouts or outages, but not with full fledged diesel engines. And while hospitals, at least those that deal with operating patients, are on battery banks and huge ass diesel engines... private small practices usually are not, if you're lucky the main server has a half broken UPS where no one ever looked after that "BATTERY FAULT" red light for a year. But the desktop computers, VPN nodes, card readers or medical equipment? If it's not something that a power outage could ruin (such as a MRT), it's probably not even battery backed. There's a German saying "the emperor is naked, he has no clothes". When it comes to the resilience of our healthcare infrastructure, the emperor isn't just naked, the emperor's skin is rotting away.
      - nradov 1y ago
        Starlink
      - marcosdumay 1y ago
        Just pointing out that none of those non-data-center buildings are data centers. I really doubt the classification of "data centers and other large, non-critical power consumers" extends to telecom infrastructure.
        mschuster91 1y ago
        > I really doubt the classification of "data centers and other large, non-critical power consumers" extends to telecom infrastructure. That's the point. Okay, cool, the datacenter is highly available, multiple power and data feeds, 24/7/365. But that highly available datacenter is useless when it cannot be reached because its data feed elements or the clients don't have power.
  - tw04 1y ago
    If power is out everywhere for an extended period of time, they aren’t doing anything but life saving surgery. Pulling up an EMR will be near the bottom of the list of concerns.
    - vineyardmike 1y ago
      Yea, that’s not how medicine works. There are patients that are in the hospital beds for a variety of reasons, and they don’t just go home during a storm. Those people still need some level of care, even if they’re not getting XRays and basic preventative care. EMRs contain a record of when patients last took some critical but dangerous drug, what their allergies and reactions are, and many other important bits of information. When one of the patients starts to exhibit some new symptom or reaction (very stressful situation!), doctors and nurses look at the EMR to understand the best course of treatment or intervention. When the EMR goes down, doctors and nurses revert to pen and paper. It’s very slow, and requires a lot of human handoff - which, critically, they’re less practiced in.
      - tw04 1y ago
        I literally help design IT resiliency for hospitals. This is absolutely how they work and part of their disaster planning. When there is an extended power outage they stop anything but vital surgery and work off pen and paper. Which you got to after spending 3 paragraphs talking about what an EMR is for.
- lowwave 1y ago
  Nice to know that, but are the data-centers Carrington Event proof? Always wondered that, but never worked in a data-center.
- lokar 1y ago
  For any big facility there will be pretty strict EPA limits on how long you can run the generators each year.
  - Bender 1y ago
    Indeed. I have faith that Texas will find a way around such rules especially if they are being regulated into running them. A Texas company I worked for was highly proficient in maximum shrugs.
  - more_corn 1y ago
    The EPA doesn’t really have the resources to enforce that. And certainly won’t have that capability under the trump administration.
  - dylan604 1y ago
    The EPA? Are they still a thing? I doubt anyone is concerned about the EPA under current management.
    - bilbo0s 1y ago
      Which is great. Until there's new management. You can't run a business by seesaw. Best to just count on that rule being enforced and place the necessary battery backups and wind or solar in place to backstop the diesel. Then make any users who need to use those data centers eat that extra cost. There's no problem with us-east costing less than us-west, and us-texas costing most of all. That's how markets work.
      - dylan604 1y ago
        But the seesaw is what the future is going to look like. If some bit of mass voting breaks out for the next election and moves to the other party, there will be a swing back the other direction. Then the following election the masses will get upset about something and swing it back. The country is too polarized to expect anything other than seesaw policies. Unless we go full revolution and just deny the other party from ever taking charge.
- abeppu 1y ago
  I wasn't involved in the specific details but I remember being told that during the power outage from hurricane Sandy, even datacenters that had sufficient generators had trouble getting the diesel to keep them running, because everyone wanted diesel at the same time and both the supply and distribution were bottlenecked. How long can most DCs run with just the fuel onhand? Have standards around that changed over time?
  - stogot 1y ago
    I recall ranges I’ve heard from operators as 24h to upwards of 72 (rare)
  - jabart 1y ago
    At a call center, that had a whole datacenter in the basement. They had two weeks of fuel on hand at all times. Being on a border of a state, they also had a 2nd grid connection in case one failed. The whole area lost power for weeks but gym was open 24/7 which became very busy during that time.
  - stevetron 1y ago
    It seems to me that Hurricane Sandy caused an issue with fueling backup generators in a New York City datacenter, where there was a bucket-brigade of people carrying fuel in pails up flights of steps several stories.
    - abeppu 1y ago
      I remember hearing this story at the time but have forgotten all the details. I think it involved a pretty well known company? I also remember hearing that some DC in NJ got special priority in getting diesel in the following days because some federal government services were hosted there and so it was treated as a national security issue to keep them supplied.
      - kalleboo 1y ago
        It reminds me of the blog during Hurricane Katrina, with the guy carrying barrels of fuel up the stairs to the 10th floor https://en.wikipedia.org/wiki/Interdictor_(blog) https://en.wikipedia.org/wiki/Interdictor_(blog)
- chaz6 1y ago
  Are they clean enough to stay within the limits of regulations around pollution when run for long periods of time?
  - Bender 1y ago
    The generators I worked with were just massive diesel engines the size of a tractor trailer each. This was pre-def, pre-cat requirements. When I would switch on multiple load banks at once I could get a small puff of soot as the engines revved up. Provided the load was not varying heavily they would at least run clean enough that one could not see or smell any exhaust. This was in the 2000 time frame. Mind you the load tests are not real-world realistic at all and this can be further mitigated using step-start programmable PDU's in the data-center. Companies that own data-centers usually have a lot of political pull with the cities due to all the assorted taxes they pay, direct and indirect revenue and employees they bring in. Cat has since added options for hydrogen [1] but I have no idea how many people have bought them. [1] - https://www.cat.com/en_US/by-industry/electric-power/electric-power-sustainability.html https://www.cat.com/en_US/by-industry/electric-power/electri...
- f1shy 1y ago
  I get your point, but saying “is ok to cut power, they have backup” isn’t a little bit kind of deviance? For example in a hospital this kind of thinking would be totally unacceptable IMHO
  - doubled112 1y ago
    Are data centers as life or death as a hospital? Also, the data center I use has run from generator power for days after storms with NO quality of service loss. Nothing like some real world testing to remind me they have this figured out. Is a hospital in the same situation? Or is only part of the hospital on those generators?
    - f1shy 1y ago
      Of course not all, depends, but I worked in a telco; and pretty much yes. Without datacenter, no cell phones (or trunking in the case I know) so no ambulance, no police, no firefighters (or a very delayed version) We had batteries and 2 generators, and once we were minutes from blackout, as the primary generator failed, and the secondary was not dimensioned to cope with the load of AA of a 45 celsius day
  - quickthrowman 1y ago
    Hospitals are obligated by law and building code to have backup generators with multiple (at least 3) separate backup power feeds for critical branch, life safety branch, and equipment branch. These are defined as ‘essential loads’ by the National Electrical Code. They can all be fed from the same generator but must use separate overcurrent protection and automatic transfer switches. Critical branch is defined as loads that are used for direct patient care, plus ‘additional task lighting, receptacles, and circuits needed for effective hospital operation.’ Life safety branch is the fire alarm system and emergency lighting, plus elevator lights and controls, medgas alarms, PA/notification systems used during building evacuation, and some generator accessories. Equipment branch has some require items including OR cooling, patient room heating, and data rooms. Some hospitals will add MRIs, non-patient care HVAC, and chillers (for air conditioning) on the generator backup system as well. There’s typically a fourth system for everything else (‘normal’ power) that is not backed by a generator. Non-emergency lighting, convenience receptacles, and other non-essential loads are on this system.
- stronglikedan 1y ago
  > did not buy enough generator capacity to run everything Maybe they can cap it for most cases - we'll turn it off for n number of days max - so that companies have a target to prepare for (or maybe that was already mentioned). Of course, no one can prepare for anything, but if it's longer than that then most other things are probably also affected anyway and no one is worried about their favorite data center.
- duxup 1y ago
  When I worked in a different career I worked with big banks and did disaster recovery tests (we didn't run the tests we just had equipment that the tests sort of centered around). We'd basically cut of a data center for a weekend and they would run from another data center that was supposed to have all the data there too. We'd even move their check processing to a backup site and they'd truck full truckloads of paper checks to a backup site. They were legit tests at the offline site too too, they'd power down equipment, and power it up and we'd fix what didn't come back up. Even the data centers would be fully powered off to test. At least at that time those banks did not skimp on those big tests and it was a big effort and pretty dang well run / complete.
bloomingeek 1y ago
This is a terrible idea. We live in the digital age, so almost everything depends on data. As in all mechanical devices, generators have a tendency to either break down(Let's not forget about associated switching gear.) or become subject to maintenance cut backs instituted my buffoons in management. We cannot depend on human errors, we can depend on the electric grid, if properly handled and maintained. Depending on generators just adds another link in the failure chain. Texas is the perfect example of how not to run an electrical grid by not allowing other states to assist in an emergency.
- Workaccount2 1y ago
  Don't put all your digital services in a texas datacenter with no fallback then...
- joezydeco 1y ago
  I'm happier with Texas being independent. Why should my state brown out because a bunch of companies put data centers in the hottest part of the continent?
- mothballed 1y ago
  IIRC the reason Texas cannot get 'assistance' from other states is that the feds made it illegal to connect to most interstate grids without following their regulatory regimes. I believe Texas does connect to Mexico and possible some other regional grids although I don't really understand the exemption for those. In this case it's not really Texas 'not allowing' other states to help but the other states not allowing Texas. Conceivably federal law could be updated to remove those regulations and Texas would absolutely connect to the interstate grid at that point.
  - vel0city 1y ago
    The Texas Interconnection does tie in to surrounding grids through DC-ties. Those are limited in how much power can be sent through them and ultimately isolate the AC frequency.
  - RHSeeger 1y ago
jqpabc123 1y ago
The main problem with the Texas grid is really very simple. Their "free market" real time power auction makes no allowances for long term concerns like reliability. Any provider who spends money to address these sort of issues is immediately priced out of the market. Some things are too important to leave to the "free market".
- mothballed 1y ago
  In a free market Texas could connect to the interstate grid which would average some of the localized reliability issues.
  - MBCook 1y ago
    It’s because they want their “free market” that they don’t connect to the national grid.
    - mothballed 1y ago
      Other states/feds were happy to take the "free market" of Texas during WWII when they direly needed it and other Gulf states begged Texas to connect. It's a holier than thou thing for the other states. "Free market" buying from Texas when they need it, when they don't it's "fuck the free market" and cut texas off via legislation and refuse to provide the same assistance that Texans provided.
    - barbacoa 1y ago
      Texas is connected to the Eastern and Mexican grids and can share power through high voltage DC interconnect stations. They simply don't sync to the phase of other grids.
- bryanlarsen 1y ago
  This can be a "have your cake and eat it too" situation. California, most of Northern Europe and many other jurisdictions have a power market without this problem. They do this by also establishing a market for capacity and/or reliability.
bee_rider 1y ago
This seems like the sort of thing that ought to be negotiated into the data center’s contract. Why force the grid operator to make every data center subject to curtailment? Of course, it is easy enough to deal with curtailment for many services. But, it should be on the table, either way.
- lokar 1y ago
  Yeah, if they needed a law it was to say you must have X% of your load subject to a load shed agreement.
tzury 1y ago
Hospitals, first responders, and such, are all cloud driven operations nowadays. It is really strange to see them categorized under Data centers and other large, non-critical power consumers
- criddell 1y ago
  Whose fault is it if hospitals place critical data and services on systems for which the SLA cannot guarantee the service they require?
  - tzury 1y ago
    Google Cloud in Texas: Region: us-south1 (Dallas) Zones: us-south1-a, us-south1-b, us-south1-c Availability: Dallas, Texas is also listed as a location for Vertex AI. Azure in Texas: Region: "South Central US" (paired with "North Central US") Azure Government: Region in Austin, Texas Availability: Azure has a physical presence in Texas. Imagine a vendor picking multi zone, or even Multi cloud... This sounds like someone was trying to mitigate the "AI is eating our small town electricity" and threw the baby along with the water.
- thrance 1y ago
  Aren't there special clouds accredited for such use cases? Ones that probably won't be targeted by this sort of bills?
- mike_d 1y ago
  Hospitals are categorized as critical care facilities, regardless of how much on prem compute they have. Under national fire and electrical codes hospitals have to have at least three completely isolated power systems (these are the different colors of electrical outlets you see). The life safety branch is the most critical and powers things like fire alarms, exit signs, stairwell lighting - anything required to evacuate patients. The critical branch is patient care like ICU beds, ventilators, and operating rooms. Lastly is the equipment branch which is essential to patient care but won't kill people immediately: HVAC, kitchens, and sterilization/cleaning equipment. There is usually a fourth system for everything else, but these are the required ones. Each has its own requirements for battery and generator backup (life safety can't take more than 10 seconds to cut over), and usually the systems are wired so that they can draw power from lower priority circuits if needed.
net01 1y ago
Honestly, not that big of a deal. the price of batteries is so low per KW and the cost of generation is so low that, in the case of a major crisis, they could just keep them running. https://ourworldindata.org/grapher/average-battery-cell-price https://ourworldindata.org/grapher/average-battery-cell-pric... And prioritizing humans (home heating, food freezing, etc ) over servers is a good thing.
- scotty79 1y ago
  Sodium batteries are about to enter the market at the fraction of the cost of current tech.
  - evanjrowley 1y ago
    Recently I became aware of "heat batteries" that might also be useful for these applications. https://www.technologyreview.com/2023/10/24/1082217/heat-battery-manufacturing-facility/ https://www.technologyreview.com/2023/10/24/1082217/heat-bat...
  - latchkey 1y ago
    Unfortunately, not until the price comes down and with lithium still getting cheaper, that'll make sodium take longer. https://www.spglobal.com/automotive-insights/en/blogs/2025/06/sodium-ion-battery-technology https://www.spglobal.com/automotive-insights/en/blogs/2025/0...
    - robotnikman 1y ago
      I hope either type of battery becomes cheap enough that I can create a power bank to power my home without breaking the bank, because right now its still rather expensive to do so.
      - latchkey 1y ago
        Follow Will Prowse on YT. It's actually not that bad all things considered (and depending on how much power you need to store).
thayne 1y ago
This seems like it will dicincentivize power plants from being built in Texas, which is probably a good thing in the long run.
- sschueller 1y ago
  No, it just tells them to run on gas directly like Elon's xAI power plant. [1] No permits needed. [1] https://www.climateandcapitalmedia.com/35-gas-turbines-no-permits-elon-musks-dirty-xai-secret/ https://www.climateandcapitalmedia.com/35-gas-turbines-no-pe...
lupusreal 1y ago
Don't grid operators already have this power? That's the sort of thing they do during "rolling blackouts", isn't it?
MBCook 1y ago
Good. If people want to build massive AI data centers to pollute and waste power for theoretical gain propped up by VC money and hype cycles then let them power themselves in an emergency too. If they’re that important, shouldn’t be a problem right? AI makes so much money right? People come first. Not virtual hype driven land grabs.
lacker 1y ago
It sounds better than the northern California system where occasionally PG&E will cut off the power of random neighborhoods because the grid is overloaded.
latchkey 1y ago
[flagged]
- sumeno 1y ago
  [flagged]
  - latchkey 1y ago
    Seems to violate a bunch of guidelines, take your pick. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
    - sumeno 1y ago
      Then flag it I guess, but adding intentionally wasteful load to the power grid so that you can turn it off when the grid is overloaded is equally as silly as my example
      - latchkey 1y ago
        wasteful is your opinion. Let me give you an example in the real world. Massena Dam in upstate NY generates, a lot, of hydropower. It was built during the time when aluminum smelting was a thing and the pots could never go cold. This constant draw on the grid and the generators was actually a good thing. When they shut down the smelting, you know, cause it created multiple superfund sites, it became a lot harder to operate the Dam. A few enterprising bitcoin miners took over the old Alcoa smelter buildings and started using the power. When I went to speak with the people who ran the dam, they were super happy about it. Not only were they now receiving revenue to keep funding the operation of dam, but they were also extending the life of it because of the continuous draw. The obvious argument would be to allocate the power elsewhere. But this is upstate NY, where nobody wants to live. There just isn't enough demand for ~2GW of power up there, plus it is insanely expensive to build transmission for that amount of power to other locations. Even just the maintenance costs on the lines going the few miles to the smelters was insane. AI is probably the next best thing to take over that power, but this was all happening long before AI became the thing that it has.
w10-1 1y ago
Regulating high-demand customers does seem most reasonable. But when the Texas legislature gets involved, it's typically to build a political franchise by creating government power than can be used arbitrarily against a fixed asset. In this case, nothing prevents the grid authority from harassing the data center during the many overload periods when a power contract is coming up for renewal. Not that businesses aren't abusing the process, either. The article mentions that 80-90% of planned data centers won't be built, due to duplicate applications. They duplicate both to secure a "phantom" slot for power, and to get localities to compete with incentives for business. It's hard to plan a grid when financial speculation and gamesmanship is driving the planning. The worst aspect is that Texas corruption is explicitly being offered as a model for other states in the PJM interconnect. I wonder how much of the high economic growth rate in the 1950's-1980's came from the lack of gamesmanship and political franchises. But it's probably unrealistic to think professionals could step back from the very maximalist positions that are selecting them as leaders.
- mike_d 1y ago
  > But when the Texas legislature gets involved, it's typically to build a political franchise by creating government power than can be used arbitrarily against a fixed asset. When I saw that it only applies to datacenters using more than 75 MW, my first thought was who are they writing the rule around? I imagine we will see things like "xAI Under 75 MW Datacenter III, LLC", "xAI Under 75 MW Datacenter IV, LLC", etc.
dev_l1x_be 1y ago
Only if we had a cheap, easy to control energy source that could power the AI revolution. https://energyeducation.ca/encyclopedia/Energy_density https://energyeducation.ca/encyclopedia/Energy_density https://www.linkedin.com/posts/bjornlomborg_there-are-no-low-energy-rich-countries-activity-7252643313084735489-gr4Q/ https://www.linkedin.com/posts/bjornlomborg_there-are-no-low...
deleted 1y ago
[deleted]
leecarraher 1y ago
On the surface, cutting less essential resources during a power supply event makes sense, the ranking of essentialness seems problematic. While the decision to stop dumping megawatts of power to train a companies next gen LLM to be used for life saving/sustaining systems makes sense, it's pretty hard to implement in all but the most extreme cases. Hospital vs gpt6 training is an easy decision, but what about deciding between someone who wants to run AC at their unoccupied home vs. cutting power to a multi-day training epoch worth hundreds of thousands of dollars. It all feels very un-capitalistic, which in the US, like it or not, is how many edge cases get resolved. Right now datacenters are just the easy target, but why not Texas' numerous fracking sites, or other less desirable industries. My guess is that an injunction on the constitutionality of this will hold it up in court for a while.
loa_in_ 1y ago
Is Texas lawmaking an example of an agile lawmaking process that adapts to the ever changing landscape of needs or is it an example of a system where this will stay codified and never repeled for the foreseeable future because it was passed once?

143 points by walterbell 1y ago

As someone that used to help manage data-centers I think this is fine. I had to load test the generators quarterly. It's good to ensure the transfer switches and generators are working and nothing beats a real world load test. The data-centers have contracts with diesel fuel providers to keep the main tanks full. They can run on diesel any time the infrastructure load is high and write off the cost in their taxes check with your tax lawyer. There may even be other tax advantages being the need for generators would be compelled by the state, perhaps a tax lawyer could find ways to make a generator tech refresh get a bigger write-down, write off better noise abatement walls as usage will increase. If a company was running with scissors and did not buy enough generator capacity to run everything then they will have to get their act together now vs. later.

Shank 1y ago
This is true, but how long will firm load shed events last, and how many of them will happen? In California, when CAISO has events that lead to firm load shedding, they're predictable, rolling blackouts and everyone knows how long they'll last, and they're assigned on a customer-specific basis. You know your power will be cut, and you know it will be a certain amount of time, and you know roughly where you are in the schedule. I could see operators of datacenters in Texas wondering about this. Also, it's underrated how much critical infrastructure is dependent on datacenters running. Like, are you going to pull someone's EHR system down that serves a local hospital, while keeping the local hospital on a critical circuit?
- Bender 1y ago
  You know your power will be cut, and you know it will be a certain amount of time, and you know roughly where you are in the schedule. Knowing when the power will be cut will not help unless I am misunderstanding you. If the data-center loses power for even a minute the generators will all fire up and then every ATS will count-down and transfer in 30 seconds. Battery backup only lasts just long enough to do a quick return to service on a failed generator and even that is sketchy at best. A properly engineered data-center can run on reduced generator capacity. Some data-centers are indeed on circuits deemed to be critical but I could see regulations changing this so that they are "business critical" vs. "life support critical" and some changes could be made at substations so that data-centers could participate in shedding. I think you are right that they will be thinking about this and adding to this probably filing preemptive lawsuits to protect their business. Such changes can violate SLA contracts businesses have with power companies and Texas is very pro-business so I can not compare it to California.
  - toast0 1y ago
    > Knowing when the power will be cut will not help unless I am misunderstanding you. If the data-center loses power for even a minute the generators will all fire up and then every ATS will count-down and transfer in 30 seconds. Battery backup only lasts just long enough to do a quick return to service on a failed generator and even that is sketchy at best. A properly engineered data-center can run on reduced generator capacity. If you know when the power will be cut, you can start the generators before the cut, and depending on your equipment, you may be able to synchronize the generator(s) with the grid and switch over without hitting the batteries. I assume big datacenters are on three phase, can you switch over each phase as it crosses zero, or do you need to do them all at once?
    - Bender 1y ago
      At least in the data-centers I helped manage the inverters were in-line running 100% duty-cycle, meaning frequency sync is not required as there is no bypass. The servers never see the raw commercial power. Data-centers in the US are indeed 3-phase. FWIW the big Cats did have controllers that would maintain sync even when commercial power was gone but we did not need it. There wasn't even a way to physically merge commercial and generator power. ATS inputs and outputs were a binary choice. I know what you mean though, the generators I worked with in the military had a manual frequency sync that required slowly turning a dial and watching light bulbs that got brighter with frequency offset. Very old equipment for Mystic Star, post-WWII era equipment. 50's to 90's
      - jimmygrapes 1y ago
        Manually syncing several of the MEP012 generators was always far more stressful to me than any physical dangers!
        Bender 1y ago
        I bet. I never messed with the trailer or skid mounted generators. It sounds like you were also USAF. At least modern day noise cancelling headphones are much better. Guessing you probably have tinnitus from working on them. At least I think that is partially where mine came from.
- vel0city 1y ago
  If they cannot handle the grid power being pulled due to load shedding, they have no business handling critical applications.
  - bluGill 1y ago
    In particular there is no way you can predict when a backhoe will take our your power line. (Even if they call to get lines located - sometimes the mark is in the wrong spot, though most are they didn't call in the first place). There are lots of other ways the power can go down suddenly and nothing you can do about it except have some other redundancy.
- nradov 1y ago
  All of the major cloud EHRs run in multiple availability zones.
  - mschuster91 1y ago
    The data center might be... but are all fiber routers, amplifiers, PoPs etc. along the line from the datacenter to the hospitals and other healthcare providers backed up as well? Particularly the "last mile" often enough is only equipped with batteries to bridge over small brownouts or outages, but not with full fledged diesel engines. And while hospitals, at least those that deal with operating patients, are on battery banks and huge ass diesel engines... private small practices usually are not, if you're lucky the main server has a half broken UPS where no one ever looked after that "BATTERY FAULT" red light for a year. But the desktop computers, VPN nodes, card readers or medical equipment? If it's not something that a power outage could ruin (such as a MRT), it's probably not even battery backed. There's a German saying "the emperor is naked, he has no clothes". When it comes to the resilience of our healthcare infrastructure, the emperor isn't just naked, the emperor's skin is rotting away.
    - nradov 1y ago
      Starlink
    - marcosdumay 1y ago
      Just pointing out that none of those non-data-center buildings are data centers. I really doubt the classification of "data centers and other large, non-critical power consumers" extends to telecom infrastructure.
      - mschuster91 1y ago
        > I really doubt the classification of "data centers and other large, non-critical power consumers" extends to telecom infrastructure. That's the point. Okay, cool, the datacenter is highly available, multiple power and data feeds, 24/7/365. But that highly available datacenter is useless when it cannot be reached because its data feed elements or the clients don't have power.
- tw04 1y ago
  If power is out everywhere for an extended period of time, they aren’t doing anything but life saving surgery. Pulling up an EMR will be near the bottom of the list of concerns.
  - vineyardmike 1y ago
    Yea, that’s not how medicine works. There are patients that are in the hospital beds for a variety of reasons, and they don’t just go home during a storm. Those people still need some level of care, even if they’re not getting XRays and basic preventative care. EMRs contain a record of when patients last took some critical but dangerous drug, what their allergies and reactions are, and many other important bits of information. When one of the patients starts to exhibit some new symptom or reaction (very stressful situation!), doctors and nurses look at the EMR to understand the best course of treatment or intervention. When the EMR goes down, doctors and nurses revert to pen and paper. It’s very slow, and requires a lot of human handoff - which, critically, they’re less practiced in.
    - tw04 1y ago
      I literally help design IT resiliency for hospitals. This is absolutely how they work and part of their disaster planning. When there is an extended power outage they stop anything but vital surgery and work off pen and paper. Which you got to after spending 3 paragraphs talking about what an EMR is for.
lowwave 1y ago
Nice to know that, but are the data-centers Carrington Event proof? Always wondered that, but never worked in a data-center.
lokar 1y ago
For any big facility there will be pretty strict EPA limits on how long you can run the generators each year.
- Bender 1y ago
  Indeed. I have faith that Texas will find a way around such rules especially if they are being regulated into running them. A Texas company I worked for was highly proficient in maximum shrugs.
- more_corn 1y ago
  The EPA doesn’t really have the resources to enforce that. And certainly won’t have that capability under the trump administration.
- dylan604 1y ago
  The EPA? Are they still a thing? I doubt anyone is concerned about the EPA under current management.
  - bilbo0s 1y ago
    Which is great. Until there's new management. You can't run a business by seesaw. Best to just count on that rule being enforced and place the necessary battery backups and wind or solar in place to backstop the diesel. Then make any users who need to use those data centers eat that extra cost. There's no problem with us-east costing less than us-west, and us-texas costing most of all. That's how markets work.
    - dylan604 1y ago
      But the seesaw is what the future is going to look like. If some bit of mass voting breaks out for the next election and moves to the other party, there will be a swing back the other direction. Then the following election the masses will get upset about something and swing it back. The country is too polarized to expect anything other than seesaw policies. Unless we go full revolution and just deny the other party from ever taking charge.
abeppu 1y ago
I wasn't involved in the specific details but I remember being told that during the power outage from hurricane Sandy, even datacenters that had sufficient generators had trouble getting the diesel to keep them running, because everyone wanted diesel at the same time and both the supply and distribution were bottlenecked. How long can most DCs run with just the fuel onhand? Have standards around that changed over time?
- stogot 1y ago
  I recall ranges I’ve heard from operators as 24h to upwards of 72 (rare)
- jabart 1y ago
  At a call center, that had a whole datacenter in the basement. They had two weeks of fuel on hand at all times. Being on a border of a state, they also had a 2nd grid connection in case one failed. The whole area lost power for weeks but gym was open 24/7 which became very busy during that time.
- stevetron 1y ago
  It seems to me that Hurricane Sandy caused an issue with fueling backup generators in a New York City datacenter, where there was a bucket-brigade of people carrying fuel in pails up flights of steps several stories.
  - abeppu 1y ago
    I remember hearing this story at the time but have forgotten all the details. I think it involved a pretty well known company? I also remember hearing that some DC in NJ got special priority in getting diesel in the following days because some federal government services were hosted there and so it was treated as a national security issue to keep them supplied.
    - kalleboo 1y ago
      It reminds me of the blog during Hurricane Katrina, with the guy carrying barrels of fuel up the stairs to the 10th floor https://en.wikipedia.org/wiki/Interdictor_(blog) https://en.wikipedia.org/wiki/Interdictor_(blog)
chaz6 1y ago
Are they clean enough to stay within the limits of regulations around pollution when run for long periods of time?
- Bender 1y ago
  The generators I worked with were just massive diesel engines the size of a tractor trailer each. This was pre-def, pre-cat requirements. When I would switch on multiple load banks at once I could get a small puff of soot as the engines revved up. Provided the load was not varying heavily they would at least run clean enough that one could not see or smell any exhaust. This was in the 2000 time frame. Mind you the load tests are not real-world realistic at all and this can be further mitigated using step-start programmable PDU's in the data-center. Companies that own data-centers usually have a lot of political pull with the cities due to all the assorted taxes they pay, direct and indirect revenue and employees they bring in. Cat has since added options for hydrogen [1] but I have no idea how many people have bought them. [1] - https://www.cat.com/en_US/by-industry/electric-power/electric-power-sustainability.html https://www.cat.com/en_US/by-industry/electric-power/electri...
f1shy 1y ago
I get your point, but saying “is ok to cut power, they have backup” isn’t a little bit kind of deviance? For example in a hospital this kind of thinking would be totally unacceptable IMHO
- doubled112 1y ago
  Are data centers as life or death as a hospital? Also, the data center I use has run from generator power for days after storms with NO quality of service loss. Nothing like some real world testing to remind me they have this figured out. Is a hospital in the same situation? Or is only part of the hospital on those generators?
  - f1shy 1y ago
    Of course not all, depends, but I worked in a telco; and pretty much yes. Without datacenter, no cell phones (or trunking in the case I know) so no ambulance, no police, no firefighters (or a very delayed version) We had batteries and 2 generators, and once we were minutes from blackout, as the primary generator failed, and the secondary was not dimensioned to cope with the load of AA of a 45 celsius day
- quickthrowman 1y ago
  Hospitals are obligated by law and building code to have backup generators with multiple (at least 3) separate backup power feeds for critical branch, life safety branch, and equipment branch. These are defined as ‘essential loads’ by the National Electrical Code. They can all be fed from the same generator but must use separate overcurrent protection and automatic transfer switches. Critical branch is defined as loads that are used for direct patient care, plus ‘additional task lighting, receptacles, and circuits needed for effective hospital operation.’ Life safety branch is the fire alarm system and emergency lighting, plus elevator lights and controls, medgas alarms, PA/notification systems used during building evacuation, and some generator accessories. Equipment branch has some require items including OR cooling, patient room heating, and data rooms. Some hospitals will add MRIs, non-patient care HVAC, and chillers (for air conditioning) on the generator backup system as well. There’s typically a fourth system for everything else (‘normal’ power) that is not backed by a generator. Non-emergency lighting, convenience receptacles, and other non-essential loads are on this system.
stronglikedan 1y ago
> did not buy enough generator capacity to run everything Maybe they can cap it for most cases - we'll turn it off for n number of days max - so that companies have a target to prepare for (or maybe that was already mentioned). Of course, no one can prepare for anything, but if it's longer than that then most other things are probably also affected anyway and no one is worried about their favorite data center.
duxup 1y ago
When I worked in a different career I worked with big banks and did disaster recovery tests (we didn't run the tests we just had equipment that the tests sort of centered around). We'd basically cut of a data center for a weekend and they would run from another data center that was supposed to have all the data there too. We'd even move their check processing to a backup site and they'd truck full truckloads of paper checks to a backup site. They were legit tests at the offline site too too, they'd power down equipment, and power it up and we'd fix what didn't come back up. Even the data centers would be fully powered off to test. At least at that time those banks did not skimp on those big tests and it was a big effort and pretty dang well run / complete.

bloomingeek 1y ago

This is a terrible idea. We live in the digital age, so almost everything depends on data. As in all mechanical devices, generators have a tendency to either break down(Let's not forget about associated switching gear.) or become subject to maintenance cut backs instituted my buffoons in management. We cannot depend on human errors, we can depend on the electric grid, if properly handled and maintained. Depending on generators just adds another link in the failure chain. Texas is the perfect example of how not to run an electrical grid by not allowing other states to assist in an emergency.

Workaccount2 1y ago
Don't put all your digital services in a texas datacenter with no fallback then...
joezydeco 1y ago
I'm happier with Texas being independent. Why should my state brown out because a bunch of companies put data centers in the hottest part of the continent?
mothballed 1y ago
IIRC the reason Texas cannot get 'assistance' from other states is that the feds made it illegal to connect to most interstate grids without following their regulatory regimes. I believe Texas does connect to Mexico and possible some other regional grids although I don't really understand the exemption for those. In this case it's not really Texas 'not allowing' other states to help but the other states not allowing Texas. Conceivably federal law could be updated to remove those regulations and Texas would absolutely connect to the interstate grid at that point.
- vel0city 1y ago
  The Texas Interconnection does tie in to surrounding grids through DC-ties. Those are limited in how much power can be sent through them and ultimately isolate the AC frequency.
- RHSeeger 1y ago

jqpabc123 1y ago

The main problem with the Texas grid is really very simple. Their "free market" real time power auction makes no allowances for long term concerns like reliability. Any provider who spends money to address these sort of issues is immediately priced out of the market. Some things are too important to leave to the "free market".

mothballed 1y ago
In a free market Texas could connect to the interstate grid which would average some of the localized reliability issues.
- MBCook 1y ago
  It’s because they want their “free market” that they don’t connect to the national grid.
  - mothballed 1y ago
    Other states/feds were happy to take the "free market" of Texas during WWII when they direly needed it and other Gulf states begged Texas to connect. It's a holier than thou thing for the other states. "Free market" buying from Texas when they need it, when they don't it's "fuck the free market" and cut texas off via legislation and refuse to provide the same assistance that Texans provided.
  - barbacoa 1y ago
    Texas is connected to the Eastern and Mexican grids and can share power through high voltage DC interconnect stations. They simply don't sync to the phase of other grids.
bryanlarsen 1y ago
This can be a "have your cake and eat it too" situation. California, most of Northern Europe and many other jurisdictions have a power market without this problem. They do this by also establishing a market for capacity and/or reliability.

bee_rider 1y ago

This seems like the sort of thing that ought to be negotiated into the data center’s contract. Why force the grid operator to make every data center subject to curtailment? Of course, it is easy enough to deal with curtailment for many services. But, it should be on the table, either way.

lokar 1y ago
Yeah, if they needed a law it was to say you must have X% of your load subject to a load shed agreement.

tzury 1y ago

Hospitals, first responders, and such, are all cloud driven operations nowadays. It is really strange to see them categorized under Data centers and other large, non-critical power consumers

criddell 1y ago
Whose fault is it if hospitals place critical data and services on systems for which the SLA cannot guarantee the service they require?
- tzury 1y ago
  Google Cloud in Texas: Region: us-south1 (Dallas) Zones: us-south1-a, us-south1-b, us-south1-c Availability: Dallas, Texas is also listed as a location for Vertex AI. Azure in Texas: Region: "South Central US" (paired with "North Central US") Azure Government: Region in Austin, Texas Availability: Azure has a physical presence in Texas. Imagine a vendor picking multi zone, or even Multi cloud... This sounds like someone was trying to mitigate the "AI is eating our small town electricity" and threw the baby along with the water.
thrance 1y ago
Aren't there special clouds accredited for such use cases? Ones that probably won't be targeted by this sort of bills?
mike_d 1y ago
Hospitals are categorized as critical care facilities, regardless of how much on prem compute they have. Under national fire and electrical codes hospitals have to have at least three completely isolated power systems (these are the different colors of electrical outlets you see). The life safety branch is the most critical and powers things like fire alarms, exit signs, stairwell lighting - anything required to evacuate patients. The critical branch is patient care like ICU beds, ventilators, and operating rooms. Lastly is the equipment branch which is essential to patient care but won't kill people immediately: HVAC, kitchens, and sterilization/cleaning equipment. There is usually a fourth system for everything else, but these are the required ones. Each has its own requirements for battery and generator backup (life safety can't take more than 10 seconds to cut over), and usually the systems are wired so that they can draw power from lower priority circuits if needed.

net01 1y ago

Honestly, not that big of a deal. the price of batteries is so low per KW and the cost of generation is so low that, in the case of a major crisis, they could just keep them running. https://ourworldindata.org/grapher/average-battery-cell-price https://ourworldindata.org/grapher/average-battery-cell-pric... And prioritizing humans (home heating, food freezing, etc ) over servers is a good thing.

scotty79 1y ago
Sodium batteries are about to enter the market at the fraction of the cost of current tech.
- evanjrowley 1y ago
  Recently I became aware of "heat batteries" that might also be useful for these applications. https://www.technologyreview.com/2023/10/24/1082217/heat-battery-manufacturing-facility/ https://www.technologyreview.com/2023/10/24/1082217/heat-bat...
- latchkey 1y ago
  Unfortunately, not until the price comes down and with lithium still getting cheaper, that'll make sodium take longer. https://www.spglobal.com/automotive-insights/en/blogs/2025/06/sodium-ion-battery-technology https://www.spglobal.com/automotive-insights/en/blogs/2025/0...
  - robotnikman 1y ago
    I hope either type of battery becomes cheap enough that I can create a power bank to power my home without breaking the bank, because right now its still rather expensive to do so.
    - latchkey 1y ago
      Follow Will Prowse on YT. It's actually not that bad all things considered (and depending on how much power you need to store).

thayne 1y ago

This seems like it will dicincentivize power plants from being built in Texas, which is probably a good thing in the long run.

sschueller 1y ago
No, it just tells them to run on gas directly like Elon's xAI power plant. [1] No permits needed. [1] https://www.climateandcapitalmedia.com/35-gas-turbines-no-permits-elon-musks-dirty-xai-secret/ https://www.climateandcapitalmedia.com/35-gas-turbines-no-pe...

lupusreal 1y ago

Don't grid operators already have this power? That's the sort of thing they do during "rolling blackouts", isn't it?

MBCook 1y ago

Good. If people want to build massive AI data centers to pollute and waste power for theoretical gain propped up by VC money and hype cycles then let them power themselves in an emergency too. If they’re that important, shouldn’t be a problem right? AI makes so much money right? People come first. Not virtual hype driven land grabs.

lacker 1y ago

It sounds better than the northern California system where occasionally PG&E will cut off the power of random neighborhoods because the grid is overloaded.

latchkey 1y ago

[flagged]

sumeno 1y ago
[flagged]
- latchkey 1y ago
  Seems to violate a bunch of guidelines, take your pick. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - sumeno 1y ago
    Then flag it I guess, but adding intentionally wasteful load to the power grid so that you can turn it off when the grid is overloaded is equally as silly as my example
    - latchkey 1y ago
      wasteful is your opinion. Let me give you an example in the real world. Massena Dam in upstate NY generates, a lot, of hydropower. It was built during the time when aluminum smelting was a thing and the pots could never go cold. This constant draw on the grid and the generators was actually a good thing. When they shut down the smelting, you know, cause it created multiple superfund sites, it became a lot harder to operate the Dam. A few enterprising bitcoin miners took over the old Alcoa smelter buildings and started using the power. When I went to speak with the people who ran the dam, they were super happy about it. Not only were they now receiving revenue to keep funding the operation of dam, but they were also extending the life of it because of the continuous draw. The obvious argument would be to allocate the power elsewhere. But this is upstate NY, where nobody wants to live. There just isn't enough demand for ~2GW of power up there, plus it is insanely expensive to build transmission for that amount of power to other locations. Even just the maintenance costs on the lines going the few miles to the smelters was insane. AI is probably the next best thing to take over that power, but this was all happening long before AI became the thing that it has.

w10-1 1y ago

Regulating high-demand customers does seem most reasonable. But when the Texas legislature gets involved, it's typically to build a political franchise by creating government power than can be used arbitrarily against a fixed asset. In this case, nothing prevents the grid authority from harassing the data center during the many overload periods when a power contract is coming up for renewal. Not that businesses aren't abusing the process, either. The article mentions that 80-90% of planned data centers won't be built, due to duplicate applications. They duplicate both to secure a "phantom" slot for power, and to get localities to compete with incentives for business. It's hard to plan a grid when financial speculation and gamesmanship is driving the planning. The worst aspect is that Texas corruption is explicitly being offered as a model for other states in the PJM interconnect. I wonder how much of the high economic growth rate in the 1950's-1980's came from the lack of gamesmanship and political franchises. But it's probably unrealistic to think professionals could step back from the very maximalist positions that are selecting them as leaders.

mike_d 1y ago
> But when the Texas legislature gets involved, it's typically to build a political franchise by creating government power than can be used arbitrarily against a fixed asset. When I saw that it only applies to datacenters using more than 75 MW, my first thought was who are they writing the rule around? I imagine we will see things like "xAI Under 75 MW Datacenter III, LLC", "xAI Under 75 MW Datacenter IV, LLC", etc.

dev_l1x_be 1y ago

Only if we had a cheap, easy to control energy source that could power the AI revolution. https://energyeducation.ca/encyclopedia/Energy_density https://energyeducation.ca/encyclopedia/Energy_density https://www.linkedin.com/posts/bjornlomborg_there-are-no-low-energy-rich-countries-activity-7252643313084735489-gr4Q/ https://www.linkedin.com/posts/bjornlomborg_there-are-no-low...

deleted 1y ago

[deleted]

leecarraher 1y ago

On the surface, cutting less essential resources during a power supply event makes sense, the ranking of essentialness seems problematic. While the decision to stop dumping megawatts of power to train a companies next gen LLM to be used for life saving/sustaining systems makes sense, it's pretty hard to implement in all but the most extreme cases. Hospital vs gpt6 training is an easy decision, but what about deciding between someone who wants to run AC at their unoccupied home vs. cutting power to a multi-day training epoch worth hundreds of thousands of dollars. It all feels very un-capitalistic, which in the US, like it or not, is how many edge cases get resolved. Right now datacenters are just the easy target, but why not Texas' numerous fracking sites, or other less desirable industries. My guess is that an injunction on the constitutionality of this will hold it up in court for a while.

loa_in_ 1y ago

Is Texas lawmaking an example of an agile lawmaking process that adapts to the ever changing landscape of needs or is it an example of a system where this will stay codified and never repeled for the foreseeable future because it was passed once?