4 ms·

Long story short they give no indication as to why their data center cooling systems were unable to handle voltage changes caused by the storm and their systems

by a2tech 8y ago

Long story short they give no indication as to why their data center cooling systems were unable to handle voltage changes caused by the storm and their systems are not designed for speedy restoration into another region.

chrisbolt 8y ago
Seems like there's more information in the preliminary RCA on https://azure.microsoft.com/en-us/status/history/ https://azure.microsoft.com/en-us/status/history/
- bpicolo 8y ago
  > Initially, the datacenter was able to maintain its operational temperatures through a load dependent thermal buffer that was designed within the cooling system. However, once this thermal buffer was depleted the datacenter temperature exceeded safe operational thresholds, and an automated shutdown of devices was initiated
  - souterrain 8y ago
    Total failure of a data center's cooling apparatus seems to be a very rare occurrence to me, perhaps limited to simultaneous failure of utility and genset power (example: electrical switchgear and fuel pumps underwater due to flooding). Anyone have any data around how frequently such a failure occurs?
    - beh9540 8y ago
      I had the same thought. The only thing I could come up with is that it wasn't a failure of power supply, but that a surge took down enough cooling systems that they couldn't maintain temperature. A lot of DC's I've seen are N+1 with cooling (or even 2N), but they all run at the same time and are the same units. Or the control system went down, and they weren't able to get it back up and running, although I would think they would have redundancy in that case.
outworlder 8y ago
> Long story short they give no indication as to why their data center cooling systems were unable to handle voltage changes caused by the storm This shouldn't matter, only to them and whoever operates the datacenter. > and their systems are not designed for speedy restoration into another region. This is what matters. You have a cloud service that is not really designed for the cloud.
gfo 8y ago
Data Center problems are Azure's issue, not the VSTS team, so your criticism should be focused there. Even though Microsoft runs all of these services that's out of scope for a VSTS blog. They linked to the Azure service details in the Azure Status link if you wanted more information: https://azure.microsoft.com/en-us/status/history/ https://azure.microsoft.com/en-us/status/history/. It sounds as though they're compiling a report on this problem which will be released in the coming weeks.

by a2tech 8y ago

chrisbolt 8y ago

Seems like there's more information in the preliminary RCA on https://azure.microsoft.com/en-us/status/history/ https://azure.microsoft.com/en-us/status/history/

bpicolo 8y ago
> Initially, the datacenter was able to maintain its operational temperatures through a load dependent thermal buffer that was designed within the cooling system. However, once this thermal buffer was depleted the datacenter temperature exceeded safe operational thresholds, and an automated shutdown of devices was initiated
- souterrain 8y ago
  Total failure of a data center's cooling apparatus seems to be a very rare occurrence to me, perhaps limited to simultaneous failure of utility and genset power (example: electrical switchgear and fuel pumps underwater due to flooding). Anyone have any data around how frequently such a failure occurs?
  - beh9540 8y ago
    I had the same thought. The only thing I could come up with is that it wasn't a failure of power supply, but that a surge took down enough cooling systems that they couldn't maintain temperature. A lot of DC's I've seen are N+1 with cooling (or even 2N), but they all run at the same time and are the same units. Or the control system went down, and they weren't able to get it back up and running, although I would think they would have redundancy in that case.

outworlder 8y ago

> Long story short they give no indication as to why their data center cooling systems were unable to handle voltage changes caused by the storm This shouldn't matter, only to them and whoever operates the datacenter. > and their systems are not designed for speedy restoration into another region. This is what matters. You have a cloud service that is not really designed for the cloud.

gfo 8y ago

Data Center problems are Azure's issue, not the VSTS team, so your criticism should be focused there. Even though Microsoft runs all of these services that's out of scope for a VSTS blog. They linked to the Azure service details in the Azure Status link if you wanted more information: https://azure.microsoft.com/en-us/status/history/ https://azure.microsoft.com/en-us/status/history/. It sounds as though they're compiling a report on this problem which will be released in the coming weeks.