9 ms·

This reminds me that I wish DNS had some way to define a load balancing algorithm for clients to use, so browsers could make load balancing decisions. This wou

This reminds me that I wish DNS had some way to define a load balancing algorithm for clients to use, so browsers could make load balancing decisions. This would eliminate the need for virtual IP addresses, having to pass originating subnet information up recursive queries, having to remove faulty VIPs (or hosts) from DNS, etc. It is baffling to me that inside the datacenter, I can control the balancing strategy for every service-to-service transaction, but for the end user's browser, all I can do is some L3 hacks to make two routers appear as one (for failover purposes). L3 balancing would be completely unnecessary if I could just program the user agent to go to the right host, after all. The end result is unnecessary cost and complexity multiplied over a billion websites.

LinuxBender 6y ago
That is a use of SRV records [1], however it was not accepted into the HTTP protocol specification. I bring it up every time there is a new protocol version but I am too lazy to write an RFC addendum for it and hope that someone else will. Existing protocols may not be modified in this manor once ratified. Maybe HTTP/4.0? /s Some applications use SRV records for load balancing. Many VoIP and video conferencing apps do this. There is a better list on Wikipedia. _service._proto.name. TTL class SRV priority weight port target. [1] - https://en.wikipedia.org/wiki/SRV_record https://en.wikipedia.org/wiki/SRV_record
- jrockway 6y ago
  Yeah, I always liked SRV records. It seems that they proved inadequate for gRPC balancing, so there are new experiments in progress (mostly xDS).
1996 6y ago
How would it be better than round robin DNS with low TTL?
- jrockway 6y ago
  Basically, it affords you the ability to cache for longer and still end up with users able to go to your website. Right now, you can try resolving common hosts, and you will see that they often provide several hosts in response to a lookup. What the browser does with those IPs is up to the browser, the standard does not define what to do. What the administrator that sets up that record wants is "send to whichever one of these seems healthy", and some browsers do do that. Other browsers just pick one at random and report failure, so your redundancy makes the system more likely to break. What I want is a way to define what to do in this case. Maybe you want to try them all in parallel and pick the first to respond (at the TCP connection level). Maybe you want to try them sequentially. Maybe you want to open a connection to all of them and send 1/n requests to each. Right now, there is no way to know what the service intends, so the browser has to guess. And each one guesses differently. (You will notice that people like Google and Cloudflare skillfully respond with only one record with a 5 minute TTL. That is so the behavior of the browser is well defined, but it also eats their entire year of 99.999% uptime with one bad reply. Your systems had better be very reliable if DNS issues can eat a year's worth of error budget.)
  - 1996 6y ago
    > What the administrator that sets up that record wants is "send to whichever one of these seems healthy In the rrDNS, remove the A record of the hosts that fails tests, or that has a load that's too high > Maybe you want to try them all in parallel and pick the first to respond (at the TCP connection level). Something a geoIP at your DNS can do, certainly not as good as doing that in the client, but it should be decent enough. > Your systems had better be very reliable if DNS issues can eat a year's worth of error budget Or, if you aren't google or cloudflare, use a 30 to 60s TTL in rrDNS, with health checks to selectively remove IP that fail, on pools splitting your servers by region with geoIP - this way, if 1/10 of your east coast servers fail, nobody from APAC will be impacted, and only 1/10th of your US east users, and only for the TTL (I'm abstracting ISP that cache for too long, but you already mitigate a lot of the problem there) I can see how it would be easier to handle that in the browser, but you may already be able to do that with some JS to estimate the latency, then store the result in a cookie that causes a reload to www.eastcoast.yoursite.com if the user sticks to www.yoursite.com or after returning home goes to www.apac.yoursite.com while new measurement say "not optimal" and update the cookie
aeden 6y ago
FWIW, there is an IETF draft that may be suitable for addressing this: https://datatracker.ietf.org/doc/draft-ietf-dnsop-svcb-https/ https://datatracker.ietf.org/doc/draft-ietf-dnsop-svcb-https...
- jrockway 6y ago
  Aha, this sounds like exactly what I was looking for!
m3047 6y ago
DNS has its own load balancing at several levels (and several different kinds): Nameserver records (NS records) used to locate a resource are served by other nameservers. NS records are chosen from among those offered in response to a query (RRs), and should all be tried if necessary to elicit a response. The algorithm isn't strictly specified and some nameservers will shuffle the order in which they return RRs in their answers, some won't assuming the stub resolver or app will do it. The foregoing also applies to A and AAAA records (returning IP addresses for names), and this has long been used as a quick and easy form of load balancing/failover, except that it doesn't really failover very well unless your app is coded to try all of the different answers (and the stub resolver returns them to your app). Nameservers querying other nameservers (caching/recursive resolvers) are supposed to compile metrics on response times when they make upstream requests and pick the fastest upstreams once they learn them. Stub resolvers (running on your device) typically query nameservers in the order you specified them in your network config, but not always. From the foregoing, you can probably see that running a caching/recursive resolver close to your devices is supposed to be desirable, by design. So far, so far. ;-) As specified, and it's never been changed, DNS tries UDP first. "Ok", you think "that must mean it will try TCP" but that's not actually true: it only tries TCP if it receives a UDP response with TC=1 (flagged as truncated). But if there's a UDP frag and it doesn't get all the frags or never gets a UDP response at all it /never/ tries TCP. You're mixing two very different environments above: 1) a datacenter with (let's just assume) VPCs and 2) a web browser. In case #2 I'll match your ante and raise you an overloaded segment which is dropping UDP packets, in which case stuff may fail to resolve at all. Oh look, I drew a wildcard: traditionally browsers have utilized the devices stub resolver, but since they've pushed ahead with DoH they've had to implement their own. People think I'm a DNS expert (what do they know?) and I guess conventional wisdom amongst myself and my peers is that UDP should perform better than TCP but anecdotally people are claiming that DoH and DoT perform better for them than their stub resolver. "Must be your ISP messing with you" says someone, "yeah right, that's gotta be it". Me: "did you try running your own local resolver?" them: "wuut?" So here's where I confess that the experts aren't always right, because I run my own local resolver and I have the same problem: when the streaming media devices are running DNS resolution on the wifi-connected laptop sucks and if I run a TCP forwarder it starts working! (https://github.com/m3047/tcp_only_forwarder https://github.com/m3047/tcp_only_forwarder). Now to case #1, the datacenter. I hope you're running your own authoritative and caching server, and you should read about views in your server config guide; using EDNS to pass subnet info is a kludge. If you're writing datacenter apps, you should consider doing your own resolution and using TCP (try the forwarder, I dare you), and provisioning accordingly (because DNS servers assume most requests will come in via UDP). If you want load balancing "you know, like nginx" I've got news for you: BIND comes with instructions for configuring nginx as a reverse TCP proxy. Oh! Looks like I've got a straight in a single suit: nginx provides SSL termination so I've got DoT for free!
m3047 6y ago
> [...] I wish DNS had some way to define a load balancing algorithm for clients to use, so browsers could make load balancing decisions. There's actually the germ of an interesting idea in that statement. If I'm going to go to the trouble, let's say, of running a local TCP forwarder (good for the whole device), can I run a packet sniffer at the same time and watch netflows and edit the responses I return to the device based on what I see performance-wise concerning those flows? Expert me says that web sites are loaded with too much cruft and since the far end terminations are spread far and wide, there's not enough opportunity to apply that learning in any practical sense. But I could be wrong. (https://github.com/m3047/shodohflo https://github.com/m3047/shodohflo)
- m3047 6y ago
  Why doesn't a stub resolver do this, and try (plain) TCP and DoT (port 852) opportunistically while it's at it: you know, good for the whole device?
  - m3047 6y ago
    > port 852 Typo, meant 853.
    - jlgaddis 6y ago
      FWIW, you can edit your own comments for up to two hours.
jeffbee 6y ago
As DNS-over-HTTPS spreads I expect to see a lot of innovation in DNS including stapling load-distribution directions along with the RRs.

LinuxBender 6y ago

That is a use of SRV records [1], however it was not accepted into the HTTP protocol specification. I bring it up every time there is a new protocol version but I am too lazy to write an RFC addendum for it and hope that someone else will. Existing protocols may not be modified in this manor once ratified. Maybe HTTP/4.0? /s Some applications use SRV records for load balancing. Many VoIP and video conferencing apps do this. There is a better list on Wikipedia. _service._proto.name. TTL class SRV priority weight port target. [1] - https://en.wikipedia.org/wiki/SRV_record https://en.wikipedia.org/wiki/SRV_record

jrockway 6y ago
Yeah, I always liked SRV records. It seems that they proved inadequate for gRPC balancing, so there are new experiments in progress (mostly xDS).

1996 6y ago

How would it be better than round robin DNS with low TTL?

jrockway 6y ago
Basically, it affords you the ability to cache for longer and still end up with users able to go to your website. Right now, you can try resolving common hosts, and you will see that they often provide several hosts in response to a lookup. What the browser does with those IPs is up to the browser, the standard does not define what to do. What the administrator that sets up that record wants is "send to whichever one of these seems healthy", and some browsers do do that. Other browsers just pick one at random and report failure, so your redundancy makes the system more likely to break. What I want is a way to define what to do in this case. Maybe you want to try them all in parallel and pick the first to respond (at the TCP connection level). Maybe you want to try them sequentially. Maybe you want to open a connection to all of them and send 1/n requests to each. Right now, there is no way to know what the service intends, so the browser has to guess. And each one guesses differently. (You will notice that people like Google and Cloudflare skillfully respond with only one record with a 5 minute TTL. That is so the behavior of the browser is well defined, but it also eats their entire year of 99.999% uptime with one bad reply. Your systems had better be very reliable if DNS issues can eat a year's worth of error budget.)
- 1996 6y ago
  > What the administrator that sets up that record wants is "send to whichever one of these seems healthy In the rrDNS, remove the A record of the hosts that fails tests, or that has a load that's too high > Maybe you want to try them all in parallel and pick the first to respond (at the TCP connection level). Something a geoIP at your DNS can do, certainly not as good as doing that in the client, but it should be decent enough. > Your systems had better be very reliable if DNS issues can eat a year's worth of error budget Or, if you aren't google or cloudflare, use a 30 to 60s TTL in rrDNS, with health checks to selectively remove IP that fail, on pools splitting your servers by region with geoIP - this way, if 1/10 of your east coast servers fail, nobody from APAC will be impacted, and only 1/10th of your US east users, and only for the TTL (I'm abstracting ISP that cache for too long, but you already mitigate a lot of the problem there) I can see how it would be easier to handle that in the browser, but you may already be able to do that with some JS to estimate the latency, then store the result in a cookie that causes a reload to www.eastcoast.yoursite.com if the user sticks to www.yoursite.com or after returning home goes to www.apac.yoursite.com while new measurement say "not optimal" and update the cookie

aeden 6y ago

FWIW, there is an IETF draft that may be suitable for addressing this: https://datatracker.ietf.org/doc/draft-ietf-dnsop-svcb-https/ https://datatracker.ietf.org/doc/draft-ietf-dnsop-svcb-https...

jrockway 6y ago
Aha, this sounds like exactly what I was looking for!

m3047 6y ago

DNS has its own load balancing at several levels (and several different kinds): Nameserver records (NS records) used to locate a resource are served by other nameservers. NS records are chosen from among those offered in response to a query (RRs), and should all be tried if necessary to elicit a response. The algorithm isn't strictly specified and some nameservers will shuffle the order in which they return RRs in their answers, some won't assuming the stub resolver or app will do it. The foregoing also applies to A and AAAA records (returning IP addresses for names), and this has long been used as a quick and easy form of load balancing/failover, except that it doesn't really failover very well unless your app is coded to try all of the different answers (and the stub resolver returns them to your app). Nameservers querying other nameservers (caching/recursive resolvers) are supposed to compile metrics on response times when they make upstream requests and pick the fastest upstreams once they learn them. Stub resolvers (running on your device) typically query nameservers in the order you specified them in your network config, but not always. From the foregoing, you can probably see that running a caching/recursive resolver close to your devices is supposed to be desirable, by design. So far, so far. ;-) As specified, and it's never been changed, DNS tries UDP first. "Ok", you think "that must mean it will try TCP" but that's not actually true: it only tries TCP if it receives a UDP response with TC=1 (flagged as truncated). But if there's a UDP frag and it doesn't get all the frags or never gets a UDP response at all it /never/ tries TCP. You're mixing two very different environments above: 1) a datacenter with (let's just assume) VPCs and 2) a web browser. In case #2 I'll match your ante and raise you an overloaded segment which is dropping UDP packets, in which case stuff may fail to resolve at all. Oh look, I drew a wildcard: traditionally browsers have utilized the devices stub resolver, but since they've pushed ahead with DoH they've had to implement their own. People think I'm a DNS expert (what do they know?) and I guess conventional wisdom amongst myself and my peers is that UDP should perform better than TCP but anecdotally people are claiming that DoH and DoT perform better for them than their stub resolver. "Must be your ISP messing with you" says someone, "yeah right, that's gotta be it". Me: "did you try running your own local resolver?" them: "wuut?" So here's where I confess that the experts aren't always right, because I run my own local resolver and I have the same problem: when the streaming media devices are running DNS resolution on the wifi-connected laptop sucks and if I run a TCP forwarder it starts working! (https://github.com/m3047/tcp_only_forwarder https://github.com/m3047/tcp_only_forwarder). Now to case #1, the datacenter. I hope you're running your own authoritative and caching server, and you should read about views in your server config guide; using EDNS to pass subnet info is a kludge. If you're writing datacenter apps, you should consider doing your own resolution and using TCP (try the forwarder, I dare you), and provisioning accordingly (because DNS servers assume most requests will come in via UDP). If you want load balancing "you know, like nginx" I've got news for you: BIND comes with instructions for configuring nginx as a reverse TCP proxy. Oh! Looks like I've got a straight in a single suit: nginx provides SSL termination so I've got DoT for free!

m3047 6y ago

> [...] I wish DNS had some way to define a load balancing algorithm for clients to use, so browsers could make load balancing decisions. There's actually the germ of an interesting idea in that statement. If I'm going to go to the trouble, let's say, of running a local TCP forwarder (good for the whole device), can I run a packet sniffer at the same time and watch netflows and edit the responses I return to the device based on what I see performance-wise concerning those flows? Expert me says that web sites are loaded with too much cruft and since the far end terminations are spread far and wide, there's not enough opportunity to apply that learning in any practical sense. But I could be wrong. (https://github.com/m3047/shodohflo https://github.com/m3047/shodohflo)

m3047 6y ago
Why doesn't a stub resolver do this, and try (plain) TCP and DoT (port 852) opportunistically while it's at it: you know, good for the whole device?
- m3047 6y ago
  > port 852 Typo, meant 853.
  - jlgaddis 6y ago
    FWIW, you can edit your own comments for up to two hours.

jeffbee 6y ago

As DNS-over-HTTPS spreads I expect to see a lot of innovation in DNS including stapling load-distribution directions along with the RRs.