3 ms·

What I found most remarkable was how in-depth their backup systems were and how many things had to go wrong for this to become an event. This happened because

• Rantenki 14y ago
  The likelihood of these occurring in parallel is probably higher than you would expect. Backup systems are difficult to test fully since you can only do it via a full failover. Without running 100% of your systems via the backup distribution breaker, for example, you cannot verify that it can handle the load. You can only design for it. The full system was probably tested at initial install, but obviously more power was being consumed by live systems than by the original test. Entropy doesn't leave backup systems alone either, so over time they can fail even without use, and it is a surprise to everyone when the "working" backup systems fall over.
• dholowiski 14y ago
  A data center outage is (usually) like a plane crash. It doesn't happen just because of one thing going wrong. A series of things have to go wrong, including backups failing, backups of backups failing, sometimes human error, and often lots of bad luck. Just like flying on a plane, single things go wrong all the time. If you don't find out about it, it means they're doing their job.