4 ms·
That technique is called "Anycast", it's mentioned at the very end of the linked article. The problem with Anycast is it works pretty good for UDP (such as DNS
by lathiat 8y ago
That technique is called "Anycast", it's mentioned at the very end of the linked article.
The problem with Anycast is it works pretty good for UDP (such as DNS) but doing Anycast with your actual TCP connection (particularly long lived ones) can be problematic because of shifting routes, multipath, etc.. meaning that sometimes your traffic will change data-center mid-flow or possibly even with alternating packets in the case of multipath somewhere etc. So suddenly your HTTP connection data is arriving at the wrong data center and the connection breaks.
What most providers currently do (as far as I understand anyway, I'm sure there is a lot of hyper optimization) is based around using Anycast to receive the DNS lookup, and use information from that DNS lookup to send back a relevant POP locations IP.
That's not always based purely on which DC received the DNS request though, that's the most naive implementation, but most of the big CDNs optimise the returned server based on the source IP of the lookup among many other factors.
Hence why it matters for public DNS, since they are the source of the DNS request.
This article however also discussed that Anycast doesn't always route traffic where you would "expect", for various reasons that extend from commercial agreements and cost all the way to bad configuration and "no idea" you may not get routed to the nearest DC.
There is also a DNS extension, called "EDNS0" (also discussed in the article) that sends the /24 of the IPv4 IP along with the query, to allow the DNS resolver to customize their response particularly for these centralized DNS services (e.g. Google DNS supports this but only if you apply to receive them as an authoritative DNS server) - however APNIC/CloudFlare explicitly chose not to use this I believe supposedly for privacy reasons with 1.1.1.1.
- stingraycharles 8y agoVery informative answer, thanks!
- markonen 8y agoThe problem of running TCP over Anycast addressing tends to be overblown. Cloudflare, for example, does all of their client-facing HTTP traffic over Anycast and so you can consider that as existence proof for it being a viable strategy. Routes simply don't seem to shift all that frequently in practice.
- akvadrako 8y agoI used to operate an anycast DNS service and we monitored it from probes all over the world. Sometimes a random ISP in USA/Europe would start sending queries to Japan and that could last for seconds to hours. I'm not sure how often it happens, but it will certainly be enough that it will cause broken connections for a few users. I'm not sure how Cloudflare deals with it - potentially they could share the TCP connection data between datacenters when needed. Or maybe it has something to do with their how edgy their setup is - if it's mostly intra-ISP cache servers being anycasted, the routing should be more stable.
- Artemis2 8y agoAFAIK a lot of Google’s HTTPS traffic is also routed using anycast. They also offer this to GCP customers under GCLB (https://cloud.google.com/load-balancing/ https://cloud.google.com/load-balancing/). I don’t know how they deal with changes in routes.
- lumisota 8y agoSome interesting anycast stability measurements: http://tma.ifip.org/wordpress/wp-content/uploads/2017/06/tma2017_paper39.pdf http://tma.ifip.org/wordpress/wp-content/uploads/2017/06/tma...
- zzzcpan 8y agoAnycast still has enough problems, that it's a miracle some are using it more or less successfully. It's not like pure DNS-based routing, where you can throw a bunch of cheap dedicated or virtual servers at it in various countries, measure dns/http latencies in the browser and then just choose lowest latency nodes for IPs, subnets.
- chewz 8y agoThank you sir, very helpful explanation especially about EDNS0. I am using DNSCrypt except queries for some resources that are CDN based like youtube, HBOGo cdnjs etc which were forwarded to 1.1.1.1. I have changed to google 8.8.8.8 now and made sure that they can see my ECS. Thanks dig +short TXT whoami.ds.akahelp.net
- js2 8y agoPedantic side note: EDNS0[0]is the name of the DNS extension mechanism, of which “client subnet”[1] is but one use of EDNS0. 0. https://tools.ietf.org/html/rfc6891 https://tools.ietf.org/html/rfc6891 1. https://tools.ietf.org/html/rfc7871 https://tools.ietf.org/html/rfc7871