5 ms·

Knowing almost nothing about networking, isn't the way Facebook handles networking somewhat of a monolithic anti-pattern? Why is a single update responsible for

by 0xy 5y ago

Knowing almost nothing about networking, isn't the way Facebook handles networking somewhat of a monolithic anti-pattern? Why is a single update responsible for taking out multiple services and why wouldn't each product or even each region within each product have their own routes, for resiliency which can then be used to rollout changes slower? By having a large centralized and monolithic system, aren't they guaranteeing that mistakes cause huge splash damage and don't separate concerns?

tw04 5y ago
Quite the opposite. Back in the day you would've had to login dozens if not hundreds of routers individually to push the change, and it likely would've been caught after screwing up the first one. This is the result of SDN (software defined networking) and being able to push a change globally from one command. I recall major ISP's screwing up their routing tables in the past but never globally on this level.
- kanbara 5y ago
  https://www.techjuice.pk/country-blocked-youtube-globally/ https://www.techjuice.pk/country-blocked-youtube-globally/ https://www.bleepingcomputer.com/news/security/major-bgp-leak-disrupts-thousands-of-networks-globally/ https://www.bleepingcomputer.com/news/security/major-bgp-lea...
- toast0 5y ago
  https://www.itproportal.com/news/misconfigured-centurylink-database-caused-global-internet-outage/ https://www.itproportal.com/news/misconfigured-centurylink-d... https://www.bleepingcomputer.com/news/technology/ibm-cloud-global-outage-caused-by-incorrect-bgp-routing/ https://www.bleepingcomputer.com/news/technology/ibm-cloud-g... (this one isn't clear, maybe BGP hijacking, and if so, not sure who the responsible party was) https://www.catchpoint.com/blog/vodafone-idea-bgp-leak https://www.catchpoint.com/blog/vodafone-idea-bgp-leak (not sure how major this one was) You can practically search ISP bgp outage and get news about the last couple times they screwed up BGP and caused a big problem. Or service BGP and get a 50/50 chance of the service screwing up BGP or an ISP/country hijacking their routes and causing a big problem. BGP is one of the best ways to break things at scale.
andrewxdiamond 5y ago
You are correct, but this problem isn’t Facebook’s doing. This is just how BGP works. Even big players like Verizon can screw it up and break the Internet. For all it’s flaws, BGP is the piece of the Internet that truly makes it decentralized. Without it, there would be a centralized routing table of some sort.
barkingcat 5y ago
If you want 1 facebook.com entity (that in turn controls instagram, whatsapp,etc) , then I get why a single AS change would take out globally. The anti-anti-pattern would be to have country specific AS's, kind of like a franchised restaurant pattern, where each country's version of fb owns their own AS that loosely forms back into the facebook mothership org, but from infrastructure point of view points to their own set of AS netspace.
- aaronax 5y ago
  Don't AS "things" only affect the IP addresses? Surely you could have DNS records for facebook.com pointing to IP addresses in multiple ASes?
silisili 5y ago
Not an antipattern. Route announcements/withdrawals are well, serious things and should likely flow through a single source. The idea of a 'microservice' based route update makes any network person uneasy.
ptd 5y ago
One might argue that the fact that you care about things like this is why you don’t run Facebook.
spoonjim 5y ago
I mean if you own 3 independent businesses, each of which would be worth over $100 billion, and you break all of them simultaneously for an entire day, including your internal email and your badge entry systems, yes, that is definitionally an anti-pattern.
- wmf 5y ago
  One of the usual justifications for acquisitions is to save money using common infrastructure. Instagram and WhatsApp haven't been independent for a while.
- tshaddox 5y ago
  Surely “anti-pattern” doesn’t just mean “anything with negative outcomes.” Couldn’t this just be a really big mistake that isn’t indicative of an anti-pattern?
  - spoonjim 5y ago
    Single point of failure is the oldest anti-pattern in the book. Goes back 5000 years.
    - tshaddox 5y ago
      Is there evidence to suggest that redundancy could have solved this problem? If this was simply a bad configuration that was propagated (as intended) to a large number of systems, I would hardly call that a single point of failure.
      - withinboredom 5y ago
        If it was a single change, deployed once that caused this, then it’s a single point of failure.
      - greiskul 5y ago
        Yeah, I don't think redundancy would have prevented this. What could prevent this is stuff like canaries. Or automatic rollbacks for this kind of change, with the change having to be submitted in two steps, one where it is applied, and one where it is manually confirmed that the change worked (or automatically by a system outside the network). If no confirmation is given after a certain time, the system should revert to the old configuration. Of course, we can't really know for sure until/if they release exactly what caused this issue.
manquer 5y ago
BGP has to converge to a single routing table. You are effectively asking is why is there a single routing table for the internet. To put in simple terms having a single routing table is what it makes it the internet we can share, otherwise it would just be a bunch of independent networks.
- toast0 5y ago
  > BGP has to converge to a single routing table. It certainly does not. If I peer with you, neither of us (generally) announce that route to our other peers, but often announce to our customers. There are many routes that are not visible to everyone, and there is no single routing table for the internet. Each BGP speaker ends up with their own routing table, although there are a lot of similarities.
  - manquer 5y ago
    I was trying to give simple explanation for someone who said they don't know networking. BGP in this context implicitly meant external BGP . Yes, no single router necessarily sees all the routes, but all routers combined generally see the internet as a single network of networks was my point. Convergence in this context is how most ASN will resolve on where/how to route a specific ASN traffic. It is hard to peer with someone and not trust the routing table they publish, that is why Pakistan could by mistake block YouTube for everyone few years back. In this case if you peered with Facebook, and they published incorrect routing for their AS you would accept it. This doesn't mean FB couldn't have used multiple ASNs did some rolling updates etc, however without knowing what exactly fb screwed up for five hours it is hard to say what they could done differently.
- Dylan16807 5y ago
  > You are effectively asking is why is there a single routing table for the internet. No, they're asking why a single set of routers is in charge of announcing BGP routes for all of facebook. If you have multiple ASes, with independent configuration sources and independent routers broadcasting them, it's a lot harder to break everything at once.
colechristensen 5y ago
Without in depth technical details of what exactly happened and how, it’s hard to say. With what we have heard and seen the answer is probably that there are some opportunities to build in more resiliency, but ultimately the kind of failure that causes domino effects is impossible to eliminate completely. This is partially due to design limitations of BGP, and partially due to it being nearly impossible to eliminate all sources of large scale failures in any highly complex system, and increasing the uptime of a system that already has a few nines costs an additional order of magnitude for each new nine. At some point you set your risk tolerance and have catastrophic failures now and then.
mftb 5y ago
Ultimately I think you are right, but I don't think this is the right way to think about the question. It's not as though they thought they were creating a monolith on purpose by following a pattern for creating monoliths. They thought they were following best practices and building a distributed system, without single-points of failure. I think a more productive way to think about it might be what's the psychology and practices that led them to be wrong? My best guess is that at some points in the stack DNS is opaque, and at other points, particularly for trained network people, it becomes transparent (i.e. invisible) and disappears (like a mirage), so then they make both the network and physical locks dependent on it (and BGP) and... lock themselves out when it fails.

by 0xy 5y ago

tw04 5y ago

Quite the opposite. Back in the day you would've had to login dozens if not hundreds of routers individually to push the change, and it likely would've been caught after screwing up the first one. This is the result of SDN (software defined networking) and being able to push a change globally from one command. I recall major ISP's screwing up their routing tables in the past but never globally on this level.

kanbara 5y ago
https://www.techjuice.pk/country-blocked-youtube-globally/ https://www.techjuice.pk/country-blocked-youtube-globally/ https://www.bleepingcomputer.com/news/security/major-bgp-leak-disrupts-thousands-of-networks-globally/ https://www.bleepingcomputer.com/news/security/major-bgp-lea...
toast0 5y ago
https://www.itproportal.com/news/misconfigured-centurylink-database-caused-global-internet-outage/ https://www.itproportal.com/news/misconfigured-centurylink-d... https://www.bleepingcomputer.com/news/technology/ibm-cloud-global-outage-caused-by-incorrect-bgp-routing/ https://www.bleepingcomputer.com/news/technology/ibm-cloud-g... (this one isn't clear, maybe BGP hijacking, and if so, not sure who the responsible party was) https://www.catchpoint.com/blog/vodafone-idea-bgp-leak https://www.catchpoint.com/blog/vodafone-idea-bgp-leak (not sure how major this one was) You can practically search ISP bgp outage and get news about the last couple times they screwed up BGP and caused a big problem. Or service BGP and get a 50/50 chance of the service screwing up BGP or an ISP/country hijacking their routes and causing a big problem. BGP is one of the best ways to break things at scale.

andrewxdiamond 5y ago

You are correct, but this problem isn’t Facebook’s doing. This is just how BGP works. Even big players like Verizon can screw it up and break the Internet. For all it’s flaws, BGP is the piece of the Internet that truly makes it decentralized. Without it, there would be a centralized routing table of some sort.

barkingcat 5y ago

If you want 1 facebook.com entity (that in turn controls instagram, whatsapp,etc) , then I get why a single AS change would take out globally. The anti-anti-pattern would be to have country specific AS's, kind of like a franchised restaurant pattern, where each country's version of fb owns their own AS that loosely forms back into the facebook mothership org, but from infrastructure point of view points to their own set of AS netspace.

aaronax 5y ago
Don't AS "things" only affect the IP addresses? Surely you could have DNS records for facebook.com pointing to IP addresses in multiple ASes?

silisili 5y ago

Not an antipattern. Route announcements/withdrawals are well, serious things and should likely flow through a single source. The idea of a 'microservice' based route update makes any network person uneasy.

ptd 5y ago

One might argue that the fact that you care about things like this is why you don’t run Facebook.

spoonjim 5y ago

I mean if you own 3 independent businesses, each of which would be worth over $100 billion, and you break all of them simultaneously for an entire day, including your internal email and your badge entry systems, yes, that is definitionally an anti-pattern.

wmf 5y ago
One of the usual justifications for acquisitions is to save money using common infrastructure. Instagram and WhatsApp haven't been independent for a while.
tshaddox 5y ago
Surely “anti-pattern” doesn’t just mean “anything with negative outcomes.” Couldn’t this just be a really big mistake that isn’t indicative of an anti-pattern?
- spoonjim 5y ago
  Single point of failure is the oldest anti-pattern in the book. Goes back 5000 years.
  - tshaddox 5y ago
    Is there evidence to suggest that redundancy could have solved this problem? If this was simply a bad configuration that was propagated (as intended) to a large number of systems, I would hardly call that a single point of failure.
    - withinboredom 5y ago
      If it was a single change, deployed once that caused this, then it’s a single point of failure.
    - greiskul 5y ago
      Yeah, I don't think redundancy would have prevented this. What could prevent this is stuff like canaries. Or automatic rollbacks for this kind of change, with the change having to be submitted in two steps, one where it is applied, and one where it is manually confirmed that the change worked (or automatically by a system outside the network). If no confirmation is given after a certain time, the system should revert to the old configuration. Of course, we can't really know for sure until/if they release exactly what caused this issue.

manquer 5y ago

BGP has to converge to a single routing table. You are effectively asking is why is there a single routing table for the internet. To put in simple terms having a single routing table is what it makes it the internet we can share, otherwise it would just be a bunch of independent networks.

toast0 5y ago
> BGP has to converge to a single routing table. It certainly does not. If I peer with you, neither of us (generally) announce that route to our other peers, but often announce to our customers. There are many routes that are not visible to everyone, and there is no single routing table for the internet. Each BGP speaker ends up with their own routing table, although there are a lot of similarities.
- manquer 5y ago
  I was trying to give simple explanation for someone who said they don't know networking. BGP in this context implicitly meant external BGP . Yes, no single router necessarily sees all the routes, but all routers combined generally see the internet as a single network of networks was my point. Convergence in this context is how most ASN will resolve on where/how to route a specific ASN traffic. It is hard to peer with someone and not trust the routing table they publish, that is why Pakistan could by mistake block YouTube for everyone few years back. In this case if you peered with Facebook, and they published incorrect routing for their AS you would accept it. This doesn't mean FB couldn't have used multiple ASNs did some rolling updates etc, however without knowing what exactly fb screwed up for five hours it is hard to say what they could done differently.
Dylan16807 5y ago
> You are effectively asking is why is there a single routing table for the internet. No, they're asking why a single set of routers is in charge of announcing BGP routes for all of facebook. If you have multiple ASes, with independent configuration sources and independent routers broadcasting them, it's a lot harder to break everything at once.

colechristensen 5y ago

Without in depth technical details of what exactly happened and how, it’s hard to say. With what we have heard and seen the answer is probably that there are some opportunities to build in more resiliency, but ultimately the kind of failure that causes domino effects is impossible to eliminate completely. This is partially due to design limitations of BGP, and partially due to it being nearly impossible to eliminate all sources of large scale failures in any highly complex system, and increasing the uptime of a system that already has a few nines costs an additional order of magnitude for each new nine. At some point you set your risk tolerance and have catastrophic failures now and then.

mftb 5y ago

Ultimately I think you are right, but I don't think this is the right way to think about the question. It's not as though they thought they were creating a monolith on purpose by following a pattern for creating monoliths. They thought they were following best practices and building a distributed system, without single-points of failure. I think a more productive way to think about it might be what's the psychology and practices that led them to be wrong? My best guess is that at some points in the stack DNS is opaque, and at other points, particularly for trained network people, it becomes transparent (i.e. invisible) and disappears (like a mirage), so then they make both the network and physical locks dependent on it (and BGP) and... lock themselves out when it fails.