3 ms·

This was not a DNS failure, nor was it caused by a single point of failure. Their entire backbone was taken offline due to a bad configuration change that went

• treesknees 5y ago
  Depends on how you define a single point. Single point doesn't mean 1 cause, it means at some point in the chain, there is a non-redundant component that when it fails the system stops. The fact FB could remove 100% of their capacity from the backbone is a SPoF if they don't have a redundant backbone completely separated from the configuration system used to control the primary. I won't pretend to know the details of FB's infrastructure, and I'm sure there was a cascade of problems that made the outage harder to fix, but it sounds like years of separate teams building on top of other layers that were "guaranteed" to be online finally collapsed into a spectacular fireball.
• slg 5y ago
  How is that not a single point of failure? The failure was the one bad configuration.
• mbreese 5y ago
  How is a bad configuration change that took out their entire network NOT a single point of failure? It's not like there were twenty different network engineers that typed the exact same command across their network. Ultimately it was one person who typed "enter" and submitted the bad change. And the problem wasn't even necessarily the change -- mistakes should be expected. The problem was the lack of a back-out strategy in case this happened.