4 ms·

My datacenter has 2 power channels, coming from two different power grids. They have never had a full outage, even when a substation two blocks away started on

My datacenter has 2 power channels, coming from two different power grids. They have never had a full outage, even when a substation two blocks away started on fire. Even if we lost power on one power channel, we would still be operational. Even if the backup power failed. That's the reason you use redundant power. If your "cloud provider" is charging you a lot but not providing your server redundant power from two grids to redundant PSUs, I don't think you're getting the best available design. In this case, it looks like OVH just threw all the power onto a single channel, which allowed a failover bug to bring down the datacenter.

Cyph0n 9y ago
The power grid in and of itself is a highly interconnected and redundant system. The fact that a substation fire didn't bring down your datacenter doesn't necessarily mean that the "redundant" setup was responsible. Unless the two grids are fully isolated (e.g., from two different countries), there is always going to be some overlap in the two power sources.
- kyledrake 9y ago
  The trick to building good datacenters is you make as many things redundant as possible, and it reduces the chance of one thing taking the entire datacenter down. Being connected to two external power providers is not the reason our datacenter stayed up, it's one of the many carefully designed redundancies that all add up to higher availability. When you have an A/B isolated power datacenter, you can (and my datacenter regularly does) shut off each channel to test that the backup systems are working, without worrying that a failed test will bring down literally every customer in the datacenter. That testing would have likely caught this problem at OVH if they were able to do it, but they instead decided to just toss all the power into a single channel, making it impossible to do testing like this without a decent chance of widespread outages. So they just had to assume it would work, until it didn't. Because our power channels are isolated, there's almost no difference between turning each one off separately, and then turning both of them off at the same time. So if we were to lose external power on both (which would be a history books blackout for how this DC gets its power, including an almost direct connection to a hydro-electric dam), you would have two well-tested backup power channels ready to go. Nothing is ever a guarantee, but I like my chances.
- cesarb 9y ago
  Even two different countries is no guarantee of isolation. The 2009 blackout affected both Paraguay and a large portion of Brazil. If two countries are near enough that it's feasible to get power from both, they're near enough that their power networks might be interconnected.

Cyph0n 9y ago

The power grid in and of itself is a highly interconnected and redundant system. The fact that a substation fire didn't bring down your datacenter doesn't necessarily mean that the "redundant" setup was responsible. Unless the two grids are fully isolated (e.g., from two different countries), there is always going to be some overlap in the two power sources.

kyledrake 9y ago
The trick to building good datacenters is you make as many things redundant as possible, and it reduces the chance of one thing taking the entire datacenter down. Being connected to two external power providers is not the reason our datacenter stayed up, it's one of the many carefully designed redundancies that all add up to higher availability. When you have an A/B isolated power datacenter, you can (and my datacenter regularly does) shut off each channel to test that the backup systems are working, without worrying that a failed test will bring down literally every customer in the datacenter. That testing would have likely caught this problem at OVH if they were able to do it, but they instead decided to just toss all the power into a single channel, making it impossible to do testing like this without a decent chance of widespread outages. So they just had to assume it would work, until it didn't. Because our power channels are isolated, there's almost no difference between turning each one off separately, and then turning both of them off at the same time. So if we were to lose external power on both (which would be a history books blackout for how this DC gets its power, including an almost direct connection to a hydro-electric dam), you would have two well-tested backup power channels ready to go. Nothing is ever a guarantee, but I like my chances.
cesarb 9y ago
Even two different countries is no guarantee of isolation. The 2009 blackout affected both Paraguay and a large portion of Brazil. If two countries are near enough that it's feasible to get power from both, they're near enough that their power networks might be interconnected.