5 ms·

It's interesting that the DNS-based solution is considered "trickery", when I don't really know anyone except for very networking-focused people who can explain

by sprayk 6y ago

It's interesting that the DNS-based solution is considered "trickery", when I don't really know anyone except for very networking-focused people who can explain how anycast works to achieve the same thing. While BGP is definitely not magic, it feels way more magic to me than DNS. The DNS-based solution, in comparison, seems way simpler to explain: get general location of IP of requester, send back the IP of a server in a DC closest to that location.

WatchDog 6y ago
Running any connection/TCP based service on an anycast IP seems to require a large and effective network operations team. Dealing with BGP route flapping, single connection traffic being split between different servers, is a difficult problem that requires extensive relationships among other network operators. Implementing EDNS Client Subnet, on the other hand is pretty simple.
- parliament32 6y ago
  The joke is you have to use anycast if you want any redundancy in your network. If you advertise your address space on a single upstream you're praying that they never have any issues. >BGP route flapping So what? Your traffic will go over to one of the other locations you're advertising on, and you keep ticking along merrily, with some extra latency. >single connection traffic being split between different servers That shouldn't happen. Keep in mind traffic just gets driven to your router, from there you load balance as you see fit. Unless you mean split between different ingress points -- TCP will happily renegotiate with very little overhead, if you're getting more serious issues your application sucks and should should be more resilient to connection state changes (if this is causing you problems, you're probably also dropping users who switch mobile to wifi...). Not to mention that these kinds of path cost updates are pretty rare on the real internet, you're not going to see this more often than once a month maybe.
  - WatchDog 6y ago
    I think you are overestimating the resilience of TCP, yes it will probably eventually renegotiate, but you create a horrible user experience. Have a read of some of the cloudflare engineering blogs, they give some sense of the technical challenges of deploying anycast. I wouldnt recommend a company deploy connection based anycast services, unless they are prepared to spend a lot on the engineering challenges. > Since packets will follow the shortest path, if a particular path is withdrawn then packets will find their way to the next shortest available route. For simple protocols like UDP that don't maintain state, Anycast is ideal and it has been used widely to load balance DNS for some time. At CloudFlare, we've done a significant amount of engineering to allow TCP to run across Anycast without flapping. This involves carefully adjusting routes in order to get optimal routing and also adjusting the way we handle protocol negotiation itself. While more complex to maintain than a Unicast network, the benefit is we can lose an entire data center and packets flow to the next closest facility without anyone noticing and hiccup. https://blog.cloudflare.com/cloudflares-architecture-eliminating-single-p/ https://blog.cloudflare.com/cloudflares-architecture-elimina... > WARP was experiencing so many failures because devices were switching servers much more often than we expected. If you recall, our ECMP router configuration uses a combination of (Source IP, Source Port, Destination IP, Destination Port) to match a packet to a server. Destination IP doesn’t generally change, WARP clients are always connecting to the same Anycast addresses. Similarly, Destination Port doesn’t change, we always listen on the same port for WARP traffic. The other two values, Source IP and Source Port, were changing much more frequently than we had planned. https://blog.cloudflare.com/warp-technical-challenges/ https://blog.cloudflare.com/warp-technical-challenges/
a1369209993 6y ago
> I don't really know anyone except for very networking-focused people who can explain how anycast works to achieve the same thing. I'm not familiar with the crap the IETF et al have smeared on it, but in principle, it's very simple: each network vertex maintains a mapping from (blocks of) addresses to edges (and the neighbors on the other ends of each edge) moving closer to that address, such that the directed edges for any given address form a rooted directed graph (ie a tree) where the root is the host with that address. Anycast just relaxes the requirement that the latent directed graph for a anycast address form a single tree with a single root, and instead allows multiple trees and roots forming a exact cover of the network. Updates still try to find the neighbor with the shortest round-trip time, but now there can be multiple network vertices with RTT=0 (ie multiple servers with the same anycast address).
- a1369209993 6y ago
  > but in principle, it's very simple Er, very simple provided that you're already doing doing unicast routing, that is. There are plenty of complexities, they just almost all already exist even without any anycast addresses.
  - parliament32 6y ago
    Exactly, if you're already doing routing/advertisements anycast just boils down to "advertise from multiple places at once" and it basically just works. If you're not doing your own routing this isn't a conversation for you.
parliament32 6y ago
I'm only calling it "trickery" because DNS isn't supposed to work that way. The entire systems structure of authoritative resolvers to recursive resolvers to cachers depends on a record being the same regardless of external factors like location or source IP. Meanwhile, anycast was literally designed for this purpose. And it's been around since 1993.[1] >explain how anycast works Advertise your IP space in multiple locations. Traffic will come to the location closest[2] to it. There's nothing particularly complicated about it. How the routing end of it works is up to the specific protocol you're using and whether this is external or internal... so it depends on BGP on the Internet (in the case we're talking about here), but you can also anycast on your internal network, and the routing will be handled by OSPF/IS-IS/whatever. Further, effectively every large service online today uses anycast clusters. Do you think 8.8.8.8 goes to a single server? Cloudflare IPs? Google IPs? It's all anycast. [1]https://tools.ietf.org/html/rfc1546 https://tools.ietf.org/html/rfc1546 [2]"closest" is subjective, it's not necessarily the geographically closest, but the "best" path. This is decided by the specific protocol you're using, but generally you can assign costs/metrics to various routes to prioritize/deprioritize them. This is your netops team's problem, not yours, so for all intents and purposes you'll always get the "best" path.

by sprayk 6y ago

WatchDog 6y ago

Running any connection/TCP based service on an anycast IP seems to require a large and effective network operations team. Dealing with BGP route flapping, single connection traffic being split between different servers, is a difficult problem that requires extensive relationships among other network operators. Implementing EDNS Client Subnet, on the other hand is pretty simple.

parliament32 6y ago
The joke is you have to use anycast if you want any redundancy in your network. If you advertise your address space on a single upstream you're praying that they never have any issues. >BGP route flapping So what? Your traffic will go over to one of the other locations you're advertising on, and you keep ticking along merrily, with some extra latency. >single connection traffic being split between different servers That shouldn't happen. Keep in mind traffic just gets driven to your router, from there you load balance as you see fit. Unless you mean split between different ingress points -- TCP will happily renegotiate with very little overhead, if you're getting more serious issues your application sucks and should should be more resilient to connection state changes (if this is causing you problems, you're probably also dropping users who switch mobile to wifi...). Not to mention that these kinds of path cost updates are pretty rare on the real internet, you're not going to see this more often than once a month maybe.
- WatchDog 6y ago
  I think you are overestimating the resilience of TCP, yes it will probably eventually renegotiate, but you create a horrible user experience. Have a read of some of the cloudflare engineering blogs, they give some sense of the technical challenges of deploying anycast. I wouldnt recommend a company deploy connection based anycast services, unless they are prepared to spend a lot on the engineering challenges. > Since packets will follow the shortest path, if a particular path is withdrawn then packets will find their way to the next shortest available route. For simple protocols like UDP that don't maintain state, Anycast is ideal and it has been used widely to load balance DNS for some time. At CloudFlare, we've done a significant amount of engineering to allow TCP to run across Anycast without flapping. This involves carefully adjusting routes in order to get optimal routing and also adjusting the way we handle protocol negotiation itself. While more complex to maintain than a Unicast network, the benefit is we can lose an entire data center and packets flow to the next closest facility without anyone noticing and hiccup. https://blog.cloudflare.com/cloudflares-architecture-eliminating-single-p/ https://blog.cloudflare.com/cloudflares-architecture-elimina... > WARP was experiencing so many failures because devices were switching servers much more often than we expected. If you recall, our ECMP router configuration uses a combination of (Source IP, Source Port, Destination IP, Destination Port) to match a packet to a server. Destination IP doesn’t generally change, WARP clients are always connecting to the same Anycast addresses. Similarly, Destination Port doesn’t change, we always listen on the same port for WARP traffic. The other two values, Source IP and Source Port, were changing much more frequently than we had planned. https://blog.cloudflare.com/warp-technical-challenges/ https://blog.cloudflare.com/warp-technical-challenges/

a1369209993 6y ago

> I don't really know anyone except for very networking-focused people who can explain how anycast works to achieve the same thing. I'm not familiar with the crap the IETF et al have smeared on it, but in principle, it's very simple: each network vertex maintains a mapping from (blocks of) addresses to edges (and the neighbors on the other ends of each edge) moving closer to that address, such that the directed edges for any given address form a rooted directed graph (ie a tree) where the root is the host with that address. Anycast just relaxes the requirement that the latent directed graph for a anycast address form a single tree with a single root, and instead allows multiple trees and roots forming a exact cover of the network. Updates still try to find the neighbor with the shortest round-trip time, but now there can be multiple network vertices with RTT=0 (ie multiple servers with the same anycast address).

a1369209993 6y ago
> but in principle, it's very simple Er, very simple provided that you're already doing doing unicast routing, that is. There are plenty of complexities, they just almost all already exist even without any anycast addresses.
- parliament32 6y ago
  Exactly, if you're already doing routing/advertisements anycast just boils down to "advertise from multiple places at once" and it basically just works. If you're not doing your own routing this isn't a conversation for you.

parliament32 6y ago

I'm only calling it "trickery" because DNS isn't supposed to work that way. The entire systems structure of authoritative resolvers to recursive resolvers to cachers depends on a record being the same regardless of external factors like location or source IP. Meanwhile, anycast was literally designed for this purpose. And it's been around since 1993.[1] >explain how anycast works Advertise your IP space in multiple locations. Traffic will come to the location closest[2] to it. There's nothing particularly complicated about it. How the routing end of it works is up to the specific protocol you're using and whether this is external or internal... so it depends on BGP on the Internet (in the case we're talking about here), but you can also anycast on your internal network, and the routing will be handled by OSPF/IS-IS/whatever. Further, effectively every large service online today uses anycast clusters. Do you think 8.8.8.8 goes to a single server? Cloudflare IPs? Google IPs? It's all anycast. [1]https://tools.ietf.org/html/rfc1546 https://tools.ietf.org/html/rfc1546 [2]"closest" is subjective, it's not necessarily the geographically closest, but the "best" path. This is decided by the specific protocol you're using, but generally you can assign costs/metrics to various routes to prioritize/deprioritize them. This is your netops team's problem, not yours, so for all intents and purposes you'll always get the "best" path.