4 ms·
The trick to building good datacenters is you make as many things redundant as possible, and it reduces the chance of one thing taking the entire datacenter dow
by kyledrake 9y ago
The trick to building good datacenters is you make as many things redundant as possible, and it reduces the chance of one thing taking the entire datacenter down. Being connected to two external power providers is not the reason our datacenter stayed up, it's one of the many carefully designed redundancies that all add up to higher availability.
When you have an A/B isolated power datacenter, you can (and my datacenter regularly does) shut off each channel to test that the backup systems are working, without worrying that a failed test will bring down literally every customer in the datacenter. That testing would have likely caught this problem at OVH if they were able to do it, but they instead decided to just toss all the power into a single channel, making it impossible to do testing like this without a decent chance of widespread outages. So they just had to assume it would work, until it didn't.
Because our power channels are isolated, there's almost no difference between turning each one off separately, and then turning both of them off at the same time. So if we were to lose external power on both (which would be a history books blackout for how this DC gets its power, including an almost direct connection to a hydro-electric dam), you would have two well-tested backup power channels ready to go. Nothing is ever a guarantee, but I like my chances.