4 ms·

Brownouts is probably the most proximate concern - a sudden increase in demand will draw down the system frequency in the vicinity, and if there aren't generati

by plorg 5y ago

Brownouts is probably the most proximate concern - a sudden increase in demand will draw down the system frequency in the vicinity, and if there aren't generation units close enough or with enough dispatchable capacity there's a small chance they would trip a protective breaker. A person I know on the power grid side said at one data center there were step functions when FB went down and then when it came up, equal to about 20% of the load behind the distribution transformer. That quantity is about as much as an aluminum smelter switching on or off.

deleted 5y ago
[deleted]
Johnny555 5y ago
But don't their datacenters all have backup generators? So worst case in a brownout, they fail over to generator power, then can start to flip back to utility power slowly. Or do they forgo backup generators and count on shifting traffic to a new datacenter if there's a regional power outage?
- kgermino 5y ago
  Edit to be less snarky: I assume they do have backup generators, though I don’t know. However if the sudden increase put that much load on the grid it could drop the frequency enough to blackout the entire neighborhood. That would be bad even if FB was able to keep running through it.
- dmoy 5y ago
  Ah yea I meant brownouts for other people haha. I figure Facebook can handle their own electrical stability just fine
  - ziml77 5y ago
    Is there any liability if Facebook had brought everything up at once and caused brownouts? Seems like it would be some form of negligence on their part harming a shared resource, but I don't know if there's any laws or contract terms with the power company that require them to pay if they mess up like that.
    - t0mas88 5y ago
      My girlfriend works in a large grid operator (in Europe). According to her there are lots of regulations and contracts on the grid operators about how they must handle reliability. So it's unlikely that Facebook would be liable if this took down half the country, because then it was the grid operator not living up to their agreements on reliability. There are a lot of automated fail-safes on this, and apparently larger industry (which a datacenter is as well) will get disconnected from the grid automatically in emergency situations before they drop residential areas. But in the end they will drop one by one everything they need to keep the larger grid running. It's not even a networked "smart" management system, the distribution points automatically react to voltage and frequency drops and they're set up to break some things like industry earlier than others.
  - sverhagen 5y ago
    The blackout of the northeast US and parts of Canada, in 2003 was really caused by something relatively small. Imagine Facebook, yesterday, causing some weird cascading effect on the power grid, and pulling half of the country with it...
- charwalker 5y ago
  For outages the generatos are great but I'm not sure how they assist with brownouts unless they can start instantly or are constantly running to provide a buffer. Short term they'd help but an instantaneous or unexpected massive traffic/CPU usage/user surge might pop too fast for the generators to start and kick in properly. Also, it might not be good for those big generators to start and stop over and over vs bringing infra back online in waves to limit spikes.
  - BenjiWiebe 5y ago
    Generators do usually start up very quickly. Under a minute.
    - charwalker 5y ago
      I guess a DC would have UPS/battery power on hand to cover an instantaneous brown out. Then the generators could be on and restoring battery power while running the DC.
  - Johnny555 5y ago
    For outages the generatos are great but I'm not sure how they assist with brownouts unless they can start instantly If the generators will help in an outage, why wouldn't they help in a brownout? You'd transition to generator when the voltage and/or frequency is outside of spec. You'd typically have some short-term power protection to keep your datacenter running until the generators start. I was skeptical about a datacenter that had less than 60 seconds of flywheel energy storage. But the data center manager said that if the generator doesn't start within 30 seconds, you're not going to get it started in an hour so having a huge battery stack that can power the datacenter for 15 minutes isn't going to help much.
bigiain 5y ago
> That quantity is about as much as an aluminum smelter switching on or off. Interestingly, the mountains east of Portland OR, where all the Aluminum smelters used to be, are now full of FAANG datacenters relying on the power infrastructure (and pricing) the Aluminum industry used to use... https://www.oregonlive.com/silicon-forest/2015/10/small-town_tax_breaks_bring_si.html https://www.oregonlive.com/silicon-forest/2015/10/small-town... And Washington state too: https://www.bizjournals.com/seattle/blog/techflash/2015/11/power-shift-data-centers-to-replace-aluminum.html https://www.bizjournals.com/seattle/blog/techflash/2015/11/p...
- Cthulhu_ 5y ago
  That's pretty interesting, I'm sure those aluminium foundries would need to be careful about turning the power on as well. Tangentially related, aluminium production in the Netherlands may shut down soon; because of a sudden spike in gas prices (due to mismanagement), electricity prices have also gone up, making producing aluminium no longer cost-effective. €2400 in electricity to produce a ton of aluminium worth €2500 kinda cost effectiveness. I wouldn't be surprised if the big datacenters here will try and offload some of their workloads to datacenters elsewhere with lower energy costs. Mind you, I'm pretty sure these datacenters make long-running deals on electricity prices.

by plorg 5y ago

deleted 5y ago

[deleted]

Johnny555 5y ago

But don't their datacenters all have backup generators? So worst case in a brownout, they fail over to generator power, then can start to flip back to utility power slowly. Or do they forgo backup generators and count on shifting traffic to a new datacenter if there's a regional power outage?

kgermino 5y ago
Edit to be less snarky: I assume they do have backup generators, though I don’t know. However if the sudden increase put that much load on the grid it could drop the frequency enough to blackout the entire neighborhood. That would be bad even if FB was able to keep running through it.
dmoy 5y ago
Ah yea I meant brownouts for other people haha. I figure Facebook can handle their own electrical stability just fine
- ziml77 5y ago
  Is there any liability if Facebook had brought everything up at once and caused brownouts? Seems like it would be some form of negligence on their part harming a shared resource, but I don't know if there's any laws or contract terms with the power company that require them to pay if they mess up like that.
  - t0mas88 5y ago
    My girlfriend works in a large grid operator (in Europe). According to her there are lots of regulations and contracts on the grid operators about how they must handle reliability. So it's unlikely that Facebook would be liable if this took down half the country, because then it was the grid operator not living up to their agreements on reliability. There are a lot of automated fail-safes on this, and apparently larger industry (which a datacenter is as well) will get disconnected from the grid automatically in emergency situations before they drop residential areas. But in the end they will drop one by one everything they need to keep the larger grid running. It's not even a networked "smart" management system, the distribution points automatically react to voltage and frequency drops and they're set up to break some things like industry earlier than others.
- sverhagen 5y ago
  The blackout of the northeast US and parts of Canada, in 2003 was really caused by something relatively small. Imagine Facebook, yesterday, causing some weird cascading effect on the power grid, and pulling half of the country with it...
charwalker 5y ago
For outages the generatos are great but I'm not sure how they assist with brownouts unless they can start instantly or are constantly running to provide a buffer. Short term they'd help but an instantaneous or unexpected massive traffic/CPU usage/user surge might pop too fast for the generators to start and kick in properly. Also, it might not be good for those big generators to start and stop over and over vs bringing infra back online in waves to limit spikes.
- BenjiWiebe 5y ago
  Generators do usually start up very quickly. Under a minute.
  - charwalker 5y ago
    I guess a DC would have UPS/battery power on hand to cover an instantaneous brown out. Then the generators could be on and restoring battery power while running the DC.
- Johnny555 5y ago
  For outages the generatos are great but I'm not sure how they assist with brownouts unless they can start instantly If the generators will help in an outage, why wouldn't they help in a brownout? You'd transition to generator when the voltage and/or frequency is outside of spec. You'd typically have some short-term power protection to keep your datacenter running until the generators start. I was skeptical about a datacenter that had less than 60 seconds of flywheel energy storage. But the data center manager said that if the generator doesn't start within 30 seconds, you're not going to get it started in an hour so having a huge battery stack that can power the datacenter for 15 minutes isn't going to help much.

bigiain 5y ago

> That quantity is about as much as an aluminum smelter switching on or off. Interestingly, the mountains east of Portland OR, where all the Aluminum smelters used to be, are now full of FAANG datacenters relying on the power infrastructure (and pricing) the Aluminum industry used to use... https://www.oregonlive.com/silicon-forest/2015/10/small-town_tax_breaks_bring_si.html https://www.oregonlive.com/silicon-forest/2015/10/small-town... And Washington state too: https://www.bizjournals.com/seattle/blog/techflash/2015/11/power-shift-data-centers-to-replace-aluminum.html https://www.bizjournals.com/seattle/blog/techflash/2015/11/p...

Cthulhu_ 5y ago
That's pretty interesting, I'm sure those aluminium foundries would need to be careful about turning the power on as well. Tangentially related, aluminium production in the Netherlands may shut down soon; because of a sudden spike in gas prices (due to mismanagement), electricity prices have also gone up, making producing aluminium no longer cost-effective. €2400 in electricity to produce a ton of aluminium worth €2500 kinda cost effectiveness. I wouldn't be surprised if the big datacenters here will try and offload some of their workloads to datacenters elsewhere with lower energy costs. Mind you, I'm pretty sure these datacenters make long-running deals on electricity prices.