4 ms·

As SRE, I get a chuckle out of this article and some of the responses. Devs mess this up constantly. DNS has one job. Hostname -> IP. Nothing further. You can

As SRE, I get a chuckle out of this article and some of the responses. Devs mess this up constantly. DNS has one job. Hostname -> IP. Nothing further. You can mess with it on server side like checking to see if HTTP server is up before delivering the IP but once IP is given, the client takes over and DNS can do nothing further so behavior will be wildly inconsistent IME. Assuming DNS RR is standard where Hostname returns multiple IPs, then it's only useful for load balancing in similar latency datacenters. If you want fancy stuff like geographic load balancing or health checks, you need fancy DNS server but at end of day, you should only return single IP so client will target the endpoint you want them to connect to.

lysace 2y ago
I've never ever come up with a scenario where RR DNS is useful in the goal of achieving high availability. I'm similarly mystified. What can be useful: dynamically adjusting DNS responses depending on what DC is up. But at this point shouldn't you be doing something via BGP instead? (This is where my knowledge breaks down.)
- stackskipton 2y ago
  Yea, Anycast IP like what Cloudflare does is the best. If you want cheaper load balancing and are ok with some downtime while DNS reconfigures, DNS system that returns IP based on which Datacenter is up works. Examples of this are Route53, Azure Traffic Manager and I assume Google has solution, I just don't know what it is.
  - lysace 2y ago
    Worked on implementing a distributed-consensus driven DNS thing like 15 years ago. We had 3 DCs around the world for a very compute-intense but not very stateful service. It actually just worked without any meaningful testing on the first single DC outage. In retrospect I'm amazed.
plagiat0r 2y ago
I've implemented a custom powerdns backend that combines heathchecks, weighted probabilistic round robin, and geo DNS and it works excellent to build and auto healing CDN. It was specifically built for multi DC or multi cloud or hybrid operations that are on separate continents, with geo DNS, heathchecks and faiolver on the DNS level at the same time. When all usa servers in the WRR pool are down, or DC is down, it starts to answers the closest next set of WRR (Canada) automatically.WRR pools are dynamic and auto healing, constantly doing http heathchecks. It is also dirt cheap, like 100x cheaper as opposed to aquire provider independent IP address space and run and operate AnyCast and having 24/7 NOC teams on this AnyCast, constantly adjusting bgp communities etc. and it is not like anycast and bgp solve anything when one server is down but other works. You can't stop announcing whole prefix if you run 200 machines but only one or two are down. TTL I'm using is 30 seconds. I never shared this backed with the world, you can't test it or purchase it. But maybe some day I'll launch a route53 competitor ;)

lysace 2y ago

I've never ever come up with a scenario where RR DNS is useful in the goal of achieving high availability. I'm similarly mystified. What can be useful: dynamically adjusting DNS responses depending on what DC is up. But at this point shouldn't you be doing something via BGP instead? (This is where my knowledge breaks down.)

stackskipton 2y ago
Yea, Anycast IP like what Cloudflare does is the best. If you want cheaper load balancing and are ok with some downtime while DNS reconfigures, DNS system that returns IP based on which Datacenter is up works. Examples of this are Route53, Azure Traffic Manager and I assume Google has solution, I just don't know what it is.
- lysace 2y ago
  Worked on implementing a distributed-consensus driven DNS thing like 15 years ago. We had 3 DCs around the world for a very compute-intense but not very stateful service. It actually just worked without any meaningful testing on the first single DC outage. In retrospect I'm amazed.

plagiat0r 2y ago

I've implemented a custom powerdns backend that combines heathchecks, weighted probabilistic round robin, and geo DNS and it works excellent to build and auto healing CDN. It was specifically built for multi DC or multi cloud or hybrid operations that are on separate continents, with geo DNS, heathchecks and faiolver on the DNS level at the same time. When all usa servers in the WRR pool are down, or DC is down, it starts to answers the closest next set of WRR (Canada) automatically.WRR pools are dynamic and auto healing, constantly doing http heathchecks. It is also dirt cheap, like 100x cheaper as opposed to aquire provider independent IP address space and run and operate AnyCast and having 24/7 NOC teams on this AnyCast, constantly adjusting bgp communities etc. and it is not like anycast and bgp solve anything when one server is down but other works. You can't stop announcing whole prefix if you run 200 machines but only one or two are down. TTL I'm using is 30 seconds. I never shared this backed with the world, you can't test it or purchase it. But maybe some day I'll launch a route53 competitor ;)