4 ms·

> As you can see, all clients correctly detect it and choose an alternative server. This is the nasty key point. The reliability is decided client-side. For e

by tetha 2y ago

> As you can see, all clients correctly detect it and choose an alternative server. This is the nasty key point. The reliability is decided client-side. For example, systemd-resolved at times enacted maximum technical correctness by always returning the lowest IP address. After all, DNS-RR is not well-defined, so always returning the lowest IPs is not wrong. It got changed after some riots, but as far as I know, Debian 11 is stuck with that behavior, or was for a long time. Or, I deal with many applications with shitty or no retry behavior. They go "Oh no, I have one connection refused, gotta cancel everything, shutdown, never try again". So now 20% - 30% of all requests die in a fire. It's an acceptable solution if you have nothing else. As the article notices, if you have quality HTTP clients with a few retries configured on them (like browsers), DNS-RR is fine to find an actual load balancer with health checks and everything, which can provide a 100% success rate. But DNS-RR is no loadbalancer and loadbalancers are better.

latchkey 2y ago
> It's an acceptable solution if you have nothing else. I'd argue it isn't acceptable at all in this day and age and that there are other solutions one should pick today long before you get to the "nothing else" choice.
- toast0 2y ago
  Anycast is nice, but it's not something you can do yourself well unless you have large scale. You need to have a large number of PoPs, and direct connectivity to many/most transit providers, or you'll get weird routing. You also need to find yourself some IP ranges. And learn BGP and find providers where you can use it. DNS round robin works as long as you can manage to find two boxes to run your stuff on, and it scales pretty high too. When I was at WhatsApp, we used DNS round robin until we moved into Facebook's hosting where it was infeasible due to servers not having public addresses. Yes, mostly not browsers, but not completely browserless.
  - latchkey 2y ago
    Back in 2013, that might have been the best solution for you. But there were still plenty of headlines... https://www.wamda.com/2013/11/whatsapp-goes-down https://www.wamda.com/2013/11/whatsapp-goes-down We're talking about today. The reason why I said Anycast is cause the vast majority of people trying to solve the need for having multiple servers in multiple locations, will just use CF or any one of the various anycast based CDN providers available today.
    - toast0 2y ago
      Oh sure, we had many outages. More outages on the one service where we tried using loadbalancers because the loadbalancers would take a one hour break every 30 days (which is pretty shitty, but that was the load balancer available, unless we wanted to run a software load balancer, which didn't make any sense). We didn't have many outages due to DNS, because we had fallback ips to contact chat in our clients. Usage was down in the 24 hours after our domain was briefly hijacked (thanks Network Solutions), and I think we lost some usage when our DNS provider was DDoSed by 'angry gamers'. But when FB broke most of their load balancers, that was a much bigger outage. BGP based outages broke everything, DNS and load balancers, so no wins there.
      - latchkey 2y ago
        > We didn't have many outages due to DNS, because we had fallback ips to contact chat in our clients. Exactly! When you control the client, you don't even need DNS. Things are actually even more secure when you don't use it, nothing to DDoS or hijack. When FB broke one set of LB's, the clients should have just routed to another set of LB's, by IP.
        toast0 2y ago
        FB likes to break everything all at once anyway... And healtchecking the load balancers wasn't working either. So DNS to regional balancers was sending people to the wrong place, and the anycast ips might have worked if you were lucky, but you might have gotten a PoP that was broken. The servers behind it were fine, if you could get to one. You could push broken DNS responses, I suppose, but it's harder than breaking a load balancer.
nerdile 2y ago
It's putting reliability in the hands of the client, or whatever random caching DNS resolver they're sitting behind. It also puts failover in those same hands. If one of your regions goes down, do you want the traffic to spread evenly to your other regions? Or pile on to the next nearest neighbor? If you care what happens, then you want to retain control of your traffic management and not cede it to others.
aarmenaa 2y ago
True. On the other hand, if you control the clients and can guarantee their behavior then DNS load balancing is highly effective. A place I used to work had internal DNS servers with hundreds of millions of records with 60 second TTLs for a bespoke internal routing system that connected incoming connections from customers with the correct resources inside our network. It was actually excellent. Changing routing was as simple as doing a DDNS update, and with NOTIFY to push changes to all child servers the average delay was less than 60 seconds for full effect. This made it easy to write more complicated tools, and I wrote a control panel that could take components from a single server to a whole data center out of service at the click of a button. There were definitely some warts in that system but as those sorts of systems go it was fast, easy to introspect, and relatively bulletproof.

by tetha 2y ago

latchkey 2y ago

> It's an acceptable solution if you have nothing else. I'd argue it isn't acceptable at all in this day and age and that there are other solutions one should pick today long before you get to the "nothing else" choice.

toast0 2y ago
Anycast is nice, but it's not something you can do yourself well unless you have large scale. You need to have a large number of PoPs, and direct connectivity to many/most transit providers, or you'll get weird routing. You also need to find yourself some IP ranges. And learn BGP and find providers where you can use it. DNS round robin works as long as you can manage to find two boxes to run your stuff on, and it scales pretty high too. When I was at WhatsApp, we used DNS round robin until we moved into Facebook's hosting where it was infeasible due to servers not having public addresses. Yes, mostly not browsers, but not completely browserless.
- latchkey 2y ago
  Back in 2013, that might have been the best solution for you. But there were still plenty of headlines... https://www.wamda.com/2013/11/whatsapp-goes-down https://www.wamda.com/2013/11/whatsapp-goes-down We're talking about today. The reason why I said Anycast is cause the vast majority of people trying to solve the need for having multiple servers in multiple locations, will just use CF or any one of the various anycast based CDN providers available today.
  - toast0 2y ago
    Oh sure, we had many outages. More outages on the one service where we tried using loadbalancers because the loadbalancers would take a one hour break every 30 days (which is pretty shitty, but that was the load balancer available, unless we wanted to run a software load balancer, which didn't make any sense). We didn't have many outages due to DNS, because we had fallback ips to contact chat in our clients. Usage was down in the 24 hours after our domain was briefly hijacked (thanks Network Solutions), and I think we lost some usage when our DNS provider was DDoSed by 'angry gamers'. But when FB broke most of their load balancers, that was a much bigger outage. BGP based outages broke everything, DNS and load balancers, so no wins there.
    - latchkey 2y ago
      > We didn't have many outages due to DNS, because we had fallback ips to contact chat in our clients. Exactly! When you control the client, you don't even need DNS. Things are actually even more secure when you don't use it, nothing to DDoS or hijack. When FB broke one set of LB's, the clients should have just routed to another set of LB's, by IP.
      - toast0 2y ago
        FB likes to break everything all at once anyway... And healtchecking the load balancers wasn't working either. So DNS to regional balancers was sending people to the wrong place, and the anycast ips might have worked if you were lucky, but you might have gotten a PoP that was broken. The servers behind it were fine, if you could get to one. You could push broken DNS responses, I suppose, but it's harder than breaking a load balancer.

nerdile 2y ago

It's putting reliability in the hands of the client, or whatever random caching DNS resolver they're sitting behind. It also puts failover in those same hands. If one of your regions goes down, do you want the traffic to spread evenly to your other regions? Or pile on to the next nearest neighbor? If you care what happens, then you want to retain control of your traffic management and not cede it to others.

aarmenaa 2y ago

True. On the other hand, if you control the clients and can guarantee their behavior then DNS load balancing is highly effective. A place I used to work had internal DNS servers with hundreds of millions of records with 60 second TTLs for a bespoke internal routing system that connected incoming connections from customers with the correct resources inside our network. It was actually excellent. Changing routing was as simple as doing a DDNS update, and with NOTIFY to push changes to all child servers the average delay was less than 60 seconds for full effect. This made it easy to write more complicated tools, and I wrote a control panel that could take components from a single server to a whole data center out of service at the click of a button. There were definitely some warts in that system but as those sorts of systems go it was fast, easy to introspect, and relatively bulletproof.