6 ms·

Isn't this another example of "how does a company with 1000s of the most able employees not notice a massive risk to business continuity sitting there waiting t

Isn't this another example of "how does a company with 1000s of the most able employees not notice a massive risk to business continuity sitting there waiting to happen"? DNS hosted on your own registrar, within your own data center for a whole host of domains. Somehow integrating your office security so that it also depends on that same DNS and running BGP updates without an emergency "undo" plan? I hate to over-simplify but Single point of failure is the first thing you ever learn about Business Continuity. We shouldn't need to learn these lessons from a big mistake, that's what college is for.

fksadfji12 5y ago
Spoken like a typical MBA freshman who thinks he has all the answers lol
marcosdumay 5y ago
Nobody expects a BGP issue. It's main weapon is surprise. But, more seriously, nobody plans for BGP failures, BGP is a single system, with a distributed database across the internet, so there's no redundancy for it, and to add it up, it's nearly completely invisible, most people don't even know it's there.
- midasuni 5y ago
  I plan for a routing failure over my network, key points have a separate simpler OOB network on an independent internet handoff, with local hosts files and local username/passwords if tacacs is down.
dijit 5y ago
Not to defend facebook, but these kinds of indirect dependencies are only obvious in hindsight. And even if someone thinks of the problem there's usually some kind of failsafe (an override key or angle grinder) in place.
- tharkun__ 5y ago
  Totally agree. A lot of time, money and effort can be spent to make everything redundant and it's one single point of failure you overlooked that will break the camel's back. The point I think that we have to keep in mind here is that it seems that Facebook overlooked many many points of failure and was very sure of their own ability at doing everything and doing it perfectly all the time. Anecdote from my previous life: Large corporation you would all recognize and which I won't name for obvious reasons. They ran their own data center(s) and it wasn't a side hustle to do that. One morning I come to the office and I have lots of messages about a massive outage. One of the data centers is offline. Like completely offline. I'm a bit fuzzy on some details because it's been a while but the main story went something like: External power goes out for whatever reason (don't remember) Batteries take over Diesel generators start up Batteries keep draining Diesels are spinning All Servers shut down What happened to this massively redundant and well planned out data center? Nobody knew and again, fuzzy on the details but the report that came out a few days later was that somewhere in that system, was a non-redudant breaker/power line combo somewhere. And that breaker went flying for some reason.
  - vvoaterr 5y ago
    So, Council Bluffs Iowa GOOGLE data center, disguised as the Gable Corporation. Got it.
  - svachalek 5y ago
    This is why Netflix has a policy of routinely breaking things on purpose. If all your other practices are pretty decent, it's easy to build up false confidence based on all the disasters that you haven't actually prevented, just made unlikely.
    - stemuk 5y ago
      Facebook does in fact habe this as well, they regularly even take out entire datacenters to see if the remaining ones take over automatically.
    - 5y ago
jaywalk 5y ago
This was not a DNS failure, nor was it caused by a single point of failure. Their entire backbone was taken offline due to a bad configuration change that went out to all of their routers.
- treesknees 5y ago
  Depends on how you define a single point. Single point doesn't mean 1 cause, it means at some point in the chain, there is a non-redundant component that when it fails the system stops. The fact FB could remove 100% of their capacity from the backbone is a SPoF if they don't have a redundant backbone completely separated from the configuration system used to control the primary. I won't pretend to know the details of FB's infrastructure, and I'm sure there was a cascade of problems that made the outage harder to fix, but it sounds like years of separate teams building on top of other layers that were "guaranteed" to be online finally collapsed into a spectacular fireball.
- slg 5y ago
  How is that not a single point of failure? The failure was the one bad configuration.
- mbreese 5y ago
  How is a bad configuration change that took out their entire network NOT a single point of failure? It's not like there were twenty different network engineers that typed the exact same command across their network. Ultimately it was one person who typed "enter" and submitted the bad change. And the problem wasn't even necessarily the change -- mistakes should be expected. The problem was the lack of a back-out strategy in case this happened.
zaptheimpaler 5y ago
You can say this about pretty much any outage or mistake ever. Every single security vulnerability or jailbreak too. How could the entire computing industry with 10000s of amazing people have missed Spectre/Meltdown? In 2018, Spectre affected basically every computer system ever - desktops & mobiles, AMD/Intel/ARM and it was basically open to exploitation since speculative execution which is decades. 1000s of employees is not as big as it sounds given how much FB does and hindsight is 20/20..

fksadfji12 5y ago

Spoken like a typical MBA freshman who thinks he has all the answers lol

marcosdumay 5y ago

Nobody expects a BGP issue. It's main weapon is surprise. But, more seriously, nobody plans for BGP failures, BGP is a single system, with a distributed database across the internet, so there's no redundancy for it, and to add it up, it's nearly completely invisible, most people don't even know it's there.

midasuni 5y ago
I plan for a routing failure over my network, key points have a separate simpler OOB network on an independent internet handoff, with local hosts files and local username/passwords if tacacs is down.

dijit 5y ago

Not to defend facebook, but these kinds of indirect dependencies are only obvious in hindsight. And even if someone thinks of the problem there's usually some kind of failsafe (an override key or angle grinder) in place.

tharkun__ 5y ago
Totally agree. A lot of time, money and effort can be spent to make everything redundant and it's one single point of failure you overlooked that will break the camel's back. The point I think that we have to keep in mind here is that it seems that Facebook overlooked many many points of failure and was very sure of their own ability at doing everything and doing it perfectly all the time. Anecdote from my previous life: Large corporation you would all recognize and which I won't name for obvious reasons. They ran their own data center(s) and it wasn't a side hustle to do that. One morning I come to the office and I have lots of messages about a massive outage. One of the data centers is offline. Like completely offline. I'm a bit fuzzy on some details because it's been a while but the main story went something like: External power goes out for whatever reason (don't remember) Batteries take over Diesel generators start up Batteries keep draining Diesels are spinning All Servers shut down What happened to this massively redundant and well planned out data center? Nobody knew and again, fuzzy on the details but the report that came out a few days later was that somewhere in that system, was a non-redudant breaker/power line combo somewhere. And that breaker went flying for some reason.
- vvoaterr 5y ago
  So, Council Bluffs Iowa GOOGLE data center, disguised as the Gable Corporation. Got it.
- svachalek 5y ago
  This is why Netflix has a policy of routinely breaking things on purpose. If all your other practices are pretty decent, it's easy to build up false confidence based on all the disasters that you haven't actually prevented, just made unlikely.
  - stemuk 5y ago
    Facebook does in fact habe this as well, they regularly even take out entire datacenters to see if the remaining ones take over automatically.
  - 5y ago

jaywalk 5y ago

This was not a DNS failure, nor was it caused by a single point of failure. Their entire backbone was taken offline due to a bad configuration change that went out to all of their routers.

treesknees 5y ago
Depends on how you define a single point. Single point doesn't mean 1 cause, it means at some point in the chain, there is a non-redundant component that when it fails the system stops. The fact FB could remove 100% of their capacity from the backbone is a SPoF if they don't have a redundant backbone completely separated from the configuration system used to control the primary. I won't pretend to know the details of FB's infrastructure, and I'm sure there was a cascade of problems that made the outage harder to fix, but it sounds like years of separate teams building on top of other layers that were "guaranteed" to be online finally collapsed into a spectacular fireball.
slg 5y ago
How is that not a single point of failure? The failure was the one bad configuration.
mbreese 5y ago
How is a bad configuration change that took out their entire network NOT a single point of failure? It's not like there were twenty different network engineers that typed the exact same command across their network. Ultimately it was one person who typed "enter" and submitted the bad change. And the problem wasn't even necessarily the change -- mistakes should be expected. The problem was the lack of a back-out strategy in case this happened.

zaptheimpaler 5y ago

You can say this about pretty much any outage or mistake ever. Every single security vulnerability or jailbreak too. How could the entire computing industry with 10000s of amazing people have missed Spectre/Meltdown? In 2018, Spectre affected basically every computer system ever - desktops & mobiles, AMD/Intel/ARM and it was basically open to exploitation since speculative execution which is decades. 1000s of employees is not as big as it sounds given how much FB does and hindsight is 20/20..