3 ms·

Agreed. Remember that Azure outage in SE Australia the other week? One of Microsoft’s DCs went down because of a power surge. Flights were cancelled and you cou

Agreed. Remember that Azure outage in SE Australia the other week? One of Microsoft’s DCs went down because of a power surge. Flights were cancelled and you couldn’t get cash from an ATM. We’ve all been in a DC. Pods with big red buttons at the end that kill the power. So if we discover an AI ‘Terminal Race Condition’ we just go in to some mundane building — here in Canberra they’re in the outer suburbs near the carpet warehouses and sex shops — and someone lifts the flap and pushes the red button. Alternatively they pull four fibre cables out of a pair of Cisco 7010s. Job done. I sleep well at night.

EvanAnderson 3y ago
> One of Microsoft’s DCs went down because of a power surge. Unrelated aside for the group: Sometime in the last 20 years the term "surge" has started being applied to any manner of power malady. Frequently I hear of outages being referred to as a "surge" even though, presumably, they're opposites. Anybody else notice this?
- orbz 3y ago
  The opposite case would be a power failure, which you don’t hear about as often as there has been a lot of improvements in preventing them at a data center level from UPS’, backup generators and connections to redundant power networks. Even if all those fail, it’s a more gradual process thats impact time can be estimated and mitigated. A power surge on the other hand triggers a failsafe immediate and unexpected power cutoff, which is why you hear about it more often. I think that overwhelmingly we only really see surges anymore.
  - EvanAnderson 3y ago
    I don't do a lot of real datacenter work (most small and medium businesses) so that probably colors my perception. I hear this talk mostly from end users-- like their building has a power outage and the end users talk about the event as "a power surge".
nonameiguess 3y ago
Or just pull a Facebook and jack up the BGP config, except on purpose, so your data center stops receiving traffic even if it has power and an active physical network link.

EvanAnderson 3y ago

> One of Microsoft’s DCs went down because of a power surge. Unrelated aside for the group: Sometime in the last 20 years the term "surge" has started being applied to any manner of power malady. Frequently I hear of outages being referred to as a "surge" even though, presumably, they're opposites. Anybody else notice this?

orbz 3y ago
The opposite case would be a power failure, which you don’t hear about as often as there has been a lot of improvements in preventing them at a data center level from UPS’, backup generators and connections to redundant power networks. Even if all those fail, it’s a more gradual process thats impact time can be estimated and mitigated. A power surge on the other hand triggers a failsafe immediate and unexpected power cutoff, which is why you hear about it more often. I think that overwhelmingly we only really see surges anymore.
- EvanAnderson 3y ago
  I don't do a lot of real datacenter work (most small and medium businesses) so that probably colors my perception. I hear this talk mostly from end users-- like their building has a power outage and the end users talk about the event as "a power surge".

nonameiguess 3y ago

Or just pull a Facebook and jack up the BGP config, except on purpose, so your data center stops receiving traffic even if it has power and an active physical network link.