5 ms·

> Second, BGP routes are not that stable. While DNS requests only require a single packet to be sent in both directions, HTTP (web) requests require establishin

by dx034 9y ago

> Second, BGP routes are not that stable. While DNS requests only require a single packet to be sent in both directions, HTTP (web) requests require establishing a connection to download the content. If the route changes, the HTTP connection is broken. I thought Cloudflare uses Anycast to avoid targeted DDOS? How do they handle changing routes during HTTP requests?

chatmasta 9y ago
Anycast means there are multiple routes going to the same destination. You get the route that is the shortest path via BGP to the anycast IP (least number of BGP hops). Once you have an established TCP session via one route, it will remain established through that route, as long as that route is still the “shortest” between your IP and the anycast IP. The route will not “change” unless cloudflare changes their routing, or you change your location/IP so that a shorter route exists. Once you’ve changed your IP, you’ve already interrupted any TCP sessions anyway. You might find these two blog posts from LinkedIn to be helpful: https://engineering.linkedin.com/network-performance/tcp-over-ip-anycast-pipe-dream-or-reality https://engineering.linkedin.com/network-performance/tcp-ove... https://engineering.linkedin.com/blog/2016/04/the-joy-of-anycast--in-side-the-datacenter https://engineering.linkedin.com/blog/2016/04/the-joy-of-any...
- chousuke 9y ago
  I think it should be clarified that "destination" refers to an IP address, not an individual host. My understanding is that anycasting means a single address corresponds to multiple hosts achieved by simply advertising it from several sources with BGP, and you will often still have multiple redundant routes to any of the individual hosts behind the anycast IP because most locations will have redundant internet links. Depending on how the routing is set up, it doesn't matter if the route changes so long as you end up on the same host consistently (or one that can at least pretend it's the same host if you do some kind of fancy session mirroring, perhaps)
  - tpetry 9y ago
    Google Cloud Global Loadbalancer seems to do this fancy session mirroring because you only have one IP for the http load balancing. I am very often impressed by the GCP products.
    - chousuke 9y ago
      There are other techniques that Google's routers will most likely use to load-balance traffic transparently to multiple hosts. A relatively simple way is to hash the (source, destination) address pair of the IP packet to determine which host to forward the packet to, so it doesn't necessarily require mirroring or any state. Only seamless failover when the host fails requires the fancy tricks.
- dx034 9y ago
  > The route will not “change” unless cloudflare changes their routing, or you change your location/IP so that a shorter route exists. Once you’ve changed your IP, you’ve already interrupted any TCP sessions anyway. That's what I thought, too. But the article explicitly states this as a potential issue.
anonacct37 9y ago
There's alot of fear around the possibilities of flapping routes, but alot of real world data seems to show it doesn't happen seem to impact web traffic that often. People often mix and match anycast/Geo DNS and anycast/unicast http. Some even go a step further and, for video files, anycast to a node that 302s to it's own unicast address.
- scurvy 9y ago
  Indeed. There are also right ways to setup anycasting and wrong ways. Right way: 1-2 major Tier1 carriers across all of your PoPs with local peering for regional eyeball networks. Wrong way: Using a different set of transit carriers at each location. You really don't want that many AS paths to reach your content from a given location (3-4 is more than enough). What you're really going for with BGP anycasting is that your local ISP has a direct route to the closest PoP via exchange peering, or that the Tier1 path drop you off to the "closest" route. Transit carriers do this for a living, and they're usually quite good at figuring out route weighting inside their own network. Yes, I know Netflix does it differently but they use a lot more smart geo DNS routing than anycasting. Edit: IMHO it's also better to choose a Tier1 with a moderate sized network that values stability and performance over size. So someone like NTT over say Level3.

by dx034 9y ago

chatmasta 9y ago

Anycast means there are multiple routes going to the same destination. You get the route that is the shortest path via BGP to the anycast IP (least number of BGP hops). Once you have an established TCP session via one route, it will remain established through that route, as long as that route is still the “shortest” between your IP and the anycast IP. The route will not “change” unless cloudflare changes their routing, or you change your location/IP so that a shorter route exists. Once you’ve changed your IP, you’ve already interrupted any TCP sessions anyway. You might find these two blog posts from LinkedIn to be helpful: https://engineering.linkedin.com/network-performance/tcp-over-ip-anycast-pipe-dream-or-reality https://engineering.linkedin.com/network-performance/tcp-ove... https://engineering.linkedin.com/blog/2016/04/the-joy-of-anycast--in-side-the-datacenter https://engineering.linkedin.com/blog/2016/04/the-joy-of-any...

chousuke 9y ago
I think it should be clarified that "destination" refers to an IP address, not an individual host. My understanding is that anycasting means a single address corresponds to multiple hosts achieved by simply advertising it from several sources with BGP, and you will often still have multiple redundant routes to any of the individual hosts behind the anycast IP because most locations will have redundant internet links. Depending on how the routing is set up, it doesn't matter if the route changes so long as you end up on the same host consistently (or one that can at least pretend it's the same host if you do some kind of fancy session mirroring, perhaps)
- tpetry 9y ago
  Google Cloud Global Loadbalancer seems to do this fancy session mirroring because you only have one IP for the http load balancing. I am very often impressed by the GCP products.
  - chousuke 9y ago
    There are other techniques that Google's routers will most likely use to load-balance traffic transparently to multiple hosts. A relatively simple way is to hash the (source, destination) address pair of the IP packet to determine which host to forward the packet to, so it doesn't necessarily require mirroring or any state. Only seamless failover when the host fails requires the fancy tricks.
dx034 9y ago
> The route will not “change” unless cloudflare changes their routing, or you change your location/IP so that a shorter route exists. Once you’ve changed your IP, you’ve already interrupted any TCP sessions anyway. That's what I thought, too. But the article explicitly states this as a potential issue.

anonacct37 9y ago

There's alot of fear around the possibilities of flapping routes, but alot of real world data seems to show it doesn't happen seem to impact web traffic that often. People often mix and match anycast/Geo DNS and anycast/unicast http. Some even go a step further and, for video files, anycast to a node that 302s to it's own unicast address.

scurvy 9y ago
Indeed. There are also right ways to setup anycasting and wrong ways. Right way: 1-2 major Tier1 carriers across all of your PoPs with local peering for regional eyeball networks. Wrong way: Using a different set of transit carriers at each location. You really don't want that many AS paths to reach your content from a given location (3-4 is more than enough). What you're really going for with BGP anycasting is that your local ISP has a direct route to the closest PoP via exchange peering, or that the Tier1 path drop you off to the "closest" route. Transit carriers do this for a living, and they're usually quite good at figuring out route weighting inside their own network. Yes, I know Netflix does it differently but they use a lot more smart geo DNS routing than anycasting. Edit: IMHO it's also better to choose a Tier1 with a moderate sized network that values stability and performance over size. So someone like NTT over say Level3.