4 ms·

> "It's an amazingly simple and elegant solution that avoids using Load Balancers." When a server is down, you have a globally distributed / cached IP address

> "It's an amazingly simple and elegant solution that avoids using Load Balancers." When a server is down, you have a globally distributed / cached IP address that you can't prevent people from hitting. https://www.cloudflare.com/learning/dns/glossary/round-robin-dns/ https://www.cloudflare.com/learning/dns/glossary/round-robin...

arrty88 2y ago
The standard today is to use a relatively low TTL and to health check the members of the pool from the dns server.
- latchkey 2y ago
  That's like saying there are traffic rules in Saigon. Exact implementation of TTL, is a suggestion.
wongarsu 2y ago
An clients tested in the article behaved correctly and chose one of the reachable servers instead. Of course somebody will inevitably misconfigure their local DNS or use a bad client. Either you accept an outage for people with broken setups or you reassign the IP to a different server in the same DC.
- latchkey 2y ago
  If you know all of your clients, then you don't even need DNS. But, you don't know all of your clients. Nor do you always know your upstream DNS provider. Design for failure. Don't fabricate failure.
  - zamadatix 2y ago
    Why would knowing your clients change whether or not you want to use DNS? Even when you control all of the clients you'll almost always want to keep using DNS. A large number of services successfully achieve their failure tolerances via these kinds of DNS methods. That doesn't mean all services would or that it's always the best answer, it just means it's a path you can consider when designing for the needs of a system.
    - latchkey 2y ago
      I'm replying to the comment above. If the article picks a few clients and it happens to work, that is effectively "knowing your clients". At which point, it means you have control over the client/server relationship and if we are trying to simplify by not using load balancers, we might as well simplify things even further, and not use DNS. It is an absurd train of thought that nobody in their right mind would consider... just like using DNS-RR as a replacement for load balancing.
      - zamadatix 2y ago
        I must be having trouble following your train of thought here - many large web services like Cloudflare and Akamai serve large volumes of content through round robin DNS balancing, what's absurd about their success? They certainly don't know every client that'll ever connect to a CDN on the internet... it just happens to work almost every time anyways. That very few clients might not instantly flip over isn't always a design failure worth deploying full load balancers. I'm also still not following why the decisions for whether or not you need a load balancer are supposed to be in any way equivalent to the decisions of when using DNS would make sense or not?
        latchkey 2y ago
        We are not talking about "large web services", we are talking about small end users spinning up their own DNS-RR "solution". LWS get away with it because of Anycast... https://www.cloudflare.com/en-gb/learning/cdn/glossary/anycast-network/ https://www.cloudflare.com/en-gb/learning/cdn/glossary/anyca...
        zamadatix 2y ago
        Anycast is certainly a nice layer to add but it's not a requirement for DNS round robin to work reliably. It does save some of the concern around relying on selection of an efficiently close choice by the client though and can be a good option for failover. More directly - is there some set of common web client I've been missing for many years that just doesn't follow DNS TTLs or try alternate records? I think the article gets it right with the wish list at the end containing a Amazon Route 53-like "pull dead entries automatically" note but maybe I'm missing something else? I've used this approach (pull the dead server entries from DNS, wait for TTL) and never caught any unexpected failures during outages but maybe I haven't been looking in the right places? If you mean it's possible to design something with round-robin DNS in a way that more clients than you expect will fail then absolutely, you can do things the wrong way with most any solution. Sometimes you can be fine with a subset of clients not always working during an outage or you can be fine with a solution which provides slower failover than an active load balancer. What I'm trying to find is why round-robin DNS must always be the wrong answer in all design cases.
toast0 2y ago
Skipping an unnecessary intermediary is worth considering. Load balancing isn't without cost, and load balancers subtly (or unsubtly) messing up connections is an issue. I've also used providers where their load balancers had worse availability than our hosts. If you control the clients, it's reasonable to call the platform dns api to get a list of ips and shuffle and iterate through in an appropriate way. Even better if you have a few stablely allocated IPs you can distribute in client binaries for when DNS is broken; but DNS is often not broken and it's nice to use for operational changes without having to push new configuration/binaries everytime you update the cluster. If your clients are browsers, default behavior is ok; they usually use IPs in order, which can be problematic [1], but otherwise, they have good retry behavior: on connection refused they try another IP right away, in case of timeout, they try at least a few different IPs. It's not ideal, and I'd use a load balancer for browsers, at least to serve the initial page load if feasible, and maybe DNS RR and semi-smart client logic in JS for websockets/etc; but DNS RR is workable for a whole site too. If your clients are not browsers and not controlled by you, best of luck? I will 100% admit that sometimes you have to assume someone built their DNS caching resolver to interpret the TTL field as a number of days, rather than number of seconds. And that clients behind those resolvers will have trouble when you update DNS, but if your loadbalancer is behind a DNS name, when it needs to change addresses, you'll deal with that then, and you won't have experience. [1] one of the RFCs suggests that OS apis should sort responses by prefix match, which might make sense if IP prefixes were heirarchical as a proxy to get to a least network distance server. But in the real world, numerically adjacent /24s are often not network adjacent, but if your servers have widely disparate addresses, you may see traffic from some client ips gravitate towards numerically similar server ips.
- ectospheno 2y ago
  > I will 100% admit that sometimes you have to assume someone built their DNS caching resolver to interpret the TTL field as a number of days, rather than number of seconds. I’ve run a min ttl of 3600 on my home network for over a year. No one has complained yet.
  - toast0 2y ago
    That's only because there's no way for service operators to effectively complain when your clients continue to hit service ips for 55 minutes after you should. And if there was, we'd first yell at all the people who continue to hit service ips for weeks and months after a change... by the time we get to complaining about one home using an hour ttl, it's not a big deal.

arrty88 2y ago

The standard today is to use a relatively low TTL and to health check the members of the pool from the dns server.

latchkey 2y ago
That's like saying there are traffic rules in Saigon. Exact implementation of TTL, is a suggestion.

wongarsu 2y ago

An clients tested in the article behaved correctly and chose one of the reachable servers instead. Of course somebody will inevitably misconfigure their local DNS or use a bad client. Either you accept an outage for people with broken setups or you reassign the IP to a different server in the same DC.

latchkey 2y ago
If you know all of your clients, then you don't even need DNS. But, you don't know all of your clients. Nor do you always know your upstream DNS provider. Design for failure. Don't fabricate failure.
- zamadatix 2y ago
  Why would knowing your clients change whether or not you want to use DNS? Even when you control all of the clients you'll almost always want to keep using DNS. A large number of services successfully achieve their failure tolerances via these kinds of DNS methods. That doesn't mean all services would or that it's always the best answer, it just means it's a path you can consider when designing for the needs of a system.
  - latchkey 2y ago
    I'm replying to the comment above. If the article picks a few clients and it happens to work, that is effectively "knowing your clients". At which point, it means you have control over the client/server relationship and if we are trying to simplify by not using load balancers, we might as well simplify things even further, and not use DNS. It is an absurd train of thought that nobody in their right mind would consider... just like using DNS-RR as a replacement for load balancing.
    - zamadatix 2y ago
      I must be having trouble following your train of thought here - many large web services like Cloudflare and Akamai serve large volumes of content through round robin DNS balancing, what's absurd about their success? They certainly don't know every client that'll ever connect to a CDN on the internet... it just happens to work almost every time anyways. That very few clients might not instantly flip over isn't always a design failure worth deploying full load balancers. I'm also still not following why the decisions for whether or not you need a load balancer are supposed to be in any way equivalent to the decisions of when using DNS would make sense or not?
      - latchkey 2y ago
        We are not talking about "large web services", we are talking about small end users spinning up their own DNS-RR "solution". LWS get away with it because of Anycast... https://www.cloudflare.com/en-gb/learning/cdn/glossary/anycast-network/ https://www.cloudflare.com/en-gb/learning/cdn/glossary/anyca...
        zamadatix 2y ago
        Anycast is certainly a nice layer to add but it's not a requirement for DNS round robin to work reliably. It does save some of the concern around relying on selection of an efficiently close choice by the client though and can be a good option for failover. More directly - is there some set of common web client I've been missing for many years that just doesn't follow DNS TTLs or try alternate records? I think the article gets it right with the wish list at the end containing a Amazon Route 53-like "pull dead entries automatically" note but maybe I'm missing something else? I've used this approach (pull the dead server entries from DNS, wait for TTL) and never caught any unexpected failures during outages but maybe I haven't been looking in the right places? If you mean it's possible to design something with round-robin DNS in a way that more clients than you expect will fail then absolutely, you can do things the wrong way with most any solution. Sometimes you can be fine with a subset of clients not always working during an outage or you can be fine with a solution which provides slower failover than an active load balancer. What I'm trying to find is why round-robin DNS must always be the wrong answer in all design cases.

toast0 2y ago

Skipping an unnecessary intermediary is worth considering. Load balancing isn't without cost, and load balancers subtly (or unsubtly) messing up connections is an issue. I've also used providers where their load balancers had worse availability than our hosts. If you control the clients, it's reasonable to call the platform dns api to get a list of ips and shuffle and iterate through in an appropriate way. Even better if you have a few stablely allocated IPs you can distribute in client binaries for when DNS is broken; but DNS is often not broken and it's nice to use for operational changes without having to push new configuration/binaries everytime you update the cluster. If your clients are browsers, default behavior is ok; they usually use IPs in order, which can be problematic [1], but otherwise, they have good retry behavior: on connection refused they try another IP right away, in case of timeout, they try at least a few different IPs. It's not ideal, and I'd use a load balancer for browsers, at least to serve the initial page load if feasible, and maybe DNS RR and semi-smart client logic in JS for websockets/etc; but DNS RR is workable for a whole site too. If your clients are not browsers and not controlled by you, best of luck? I will 100% admit that sometimes you have to assume someone built their DNS caching resolver to interpret the TTL field as a number of days, rather than number of seconds. And that clients behind those resolvers will have trouble when you update DNS, but if your loadbalancer is behind a DNS name, when it needs to change addresses, you'll deal with that then, and you won't have experience. [1] one of the RFCs suggests that OS apis should sort responses by prefix match, which might make sense if IP prefixes were heirarchical as a proxy to get to a least network distance server. But in the real world, numerically adjacent /24s are often not network adjacent, but if your servers have widely disparate addresses, you may see traffic from some client ips gravitate towards numerically similar server ips.

ectospheno 2y ago
> I will 100% admit that sometimes you have to assume someone built their DNS caching resolver to interpret the TTL field as a number of days, rather than number of seconds. I’ve run a min ttl of 3600 on my home network for over a year. No one has complained yet.
- toast0 2y ago
  That's only because there's no way for service operators to effectively complain when your clients continue to hit service ips for 55 minutes after you should. And if there was, we'd first yell at all the people who continue to hit service ips for weeks and months after a change... by the time we get to complaining about one home using an hour ttl, it's not a big deal.