26 ms·

DNS based GEO load balancing/CDN's are wrong idea today. For example if you use DNS that has bad configuration or one that is not supplied by your ISP, then you

DNS based GEO load balancing/CDN's are wrong idea today. For example if you use DNS that has bad configuration or one that is not supplied by your ISP, then you could be routed to servers thousands km/miles from your location. Last time I've checked akamai used that flawed dns based system. What you want to use now is what for example cloudflare uses which is anycast IP. You just announce same IP class on multiple routers/locations and all traffic is routed to the nearest locations thanks to how BGP routing works.

toast0 9y ago
As with all things, there are tradeoffs. As you say, GEO DNS doesn't work very well when the authoritative server doesn't get enough information to give a good answer; for example, if all of a nationwide ISP uses the same pool of recursive DNS, and doesn't proved EDNS client subnet, the authoritative server will not get any signal about where in the country the users are. Or, if you use DNS servers in another country for some reason, you may be assumed to be nearby to that country. Anycast isn't perfect either though. It may be more likely to get you a route with fewer hops, but that's not necessarily the lowest latency link. And if there's a congested link in the path, it's much harder to steer a portion of users, but not all to alternate servers. You also have to deal with the potential of routing changing from packet to packet, and potentially different locations getting different parts of the same connection. It's not 'just announce the same IPs from multiple locations' and take a nap. Anyway, unless you have thousands of datacenters well distributed throughout the globe, a lot of people are going to have high latency to whatever server is chosen, and high latency and very small windows == poor throughput, regardless of how you get to a server.
- londons_explore 9y ago
  Do any big networks do anycast tcp stream reassembly when a connection from a user gets some packets split between different datacenters? One could imagine a system where one datacenter forwards the packets (encapsulated) to the other so that the stream isn't broken.
- mjevans 9y ago
  The DNS server is the wrong place to make that decision. If there is a set of choices and a decision to make, letting the client choose would be the best option. I think I'd start with a client shuffling the list of results and asking for small fragments of the dataset from each. Based on the latency, bandwidth detected, and the size of the requested data it would then pick the most optimal set of servers. This also allows for fallback in case servers are unresponsive.
- lossolo 9y ago
  > Anycast isn't perfect either though. Sure, but it's better than DNS based solution. > It may be more likely to get you a route with fewer hops, but that's not necessarily the lowest latency link. And if there's a congested link in the path, it's much harder to steer a portion of users, but not all to alternate servers. This is also true for DNS based solution because of DNS caching. Smart networks will change routes before your DNS changes get propagated. > Anyway, unless you have thousands of datacenters well distributed throughout the globe, a lot of people are going to have high latency to whatever server is chosen, and high latency and very small windows == poor throughput, regardless of how you get to a server. Depends what's your definition of high latency. If you live on desert in Africa or some forgotten village in India, then you will always have high latency because no one will build a POP there.
deleted 9y ago
[deleted]
skuhn 9y ago
A DNS based GSLB system makes a lot of sense when you don't control (or trust) the client and you need to make smaller transfers where latency is more important than throughput -- i.e. web pages being loaded by any random web browser. It's definitely not an ideal method when throughput is more important than latency, as in the case of 40GB game downloads (or streaming video, for that matter). Sony could easily implement an alternate system where DNS based GSLB is used to first locate a nearish node that then issues a redirect to an actually nearby node after doing some basic lookups (to resolve the "I am not where my DNS resolver is" problem). Sony could also implement additional logic on the client (since they control the client after all), where it obtains a list of possible sources and runs races against them before deciding. Some server control is still desirable, in the event of a business reason to prefer a less optimal path or an overloaded source. But again, since you control the client, you can have it respond to your "chill out" packets and move to its next nearest target. That Sony doesn't appear to do any of this probably speaks more to a lack of expertise than anything else. They run a pretty decent online service with PSN, but I get the sense that they aren't really hiring people with a ton of Internet content distribution experience.
- jsnell 9y ago
  Sony control the client, but not the servers. They don't run their own CDN, but use at least half a dozen different commodity CDNs. So extra server side logic won't really be an option, it's just static files over HTTP in bulk. But yes, there's a lot of client side smarts they could benefit from.

toast0 9y ago

As with all things, there are tradeoffs. As you say, GEO DNS doesn't work very well when the authoritative server doesn't get enough information to give a good answer; for example, if all of a nationwide ISP uses the same pool of recursive DNS, and doesn't proved EDNS client subnet, the authoritative server will not get any signal about where in the country the users are. Or, if you use DNS servers in another country for some reason, you may be assumed to be nearby to that country. Anycast isn't perfect either though. It may be more likely to get you a route with fewer hops, but that's not necessarily the lowest latency link. And if there's a congested link in the path, it's much harder to steer a portion of users, but not all to alternate servers. You also have to deal with the potential of routing changing from packet to packet, and potentially different locations getting different parts of the same connection. It's not 'just announce the same IPs from multiple locations' and take a nap. Anyway, unless you have thousands of datacenters well distributed throughout the globe, a lot of people are going to have high latency to whatever server is chosen, and high latency and very small windows == poor throughput, regardless of how you get to a server.

londons_explore 9y ago
Do any big networks do anycast tcp stream reassembly when a connection from a user gets some packets split between different datacenters? One could imagine a system where one datacenter forwards the packets (encapsulated) to the other so that the stream isn't broken.
mjevans 9y ago
The DNS server is the wrong place to make that decision. If there is a set of choices and a decision to make, letting the client choose would be the best option. I think I'd start with a client shuffling the list of results and asking for small fragments of the dataset from each. Based on the latency, bandwidth detected, and the size of the requested data it would then pick the most optimal set of servers. This also allows for fallback in case servers are unresponsive.
lossolo 9y ago
> Anycast isn't perfect either though. Sure, but it's better than DNS based solution. > It may be more likely to get you a route with fewer hops, but that's not necessarily the lowest latency link. And if there's a congested link in the path, it's much harder to steer a portion of users, but not all to alternate servers. This is also true for DNS based solution because of DNS caching. Smart networks will change routes before your DNS changes get propagated. > Anyway, unless you have thousands of datacenters well distributed throughout the globe, a lot of people are going to have high latency to whatever server is chosen, and high latency and very small windows == poor throughput, regardless of how you get to a server. Depends what's your definition of high latency. If you live on desert in Africa or some forgotten village in India, then you will always have high latency because no one will build a POP there.

deleted 9y ago

[deleted]

skuhn 9y ago

A DNS based GSLB system makes a lot of sense when you don't control (or trust) the client and you need to make smaller transfers where latency is more important than throughput -- i.e. web pages being loaded by any random web browser. It's definitely not an ideal method when throughput is more important than latency, as in the case of 40GB game downloads (or streaming video, for that matter). Sony could easily implement an alternate system where DNS based GSLB is used to first locate a nearish node that then issues a redirect to an actually nearby node after doing some basic lookups (to resolve the "I am not where my DNS resolver is" problem). Sony could also implement additional logic on the client (since they control the client after all), where it obtains a list of possible sources and runs races against them before deciding. Some server control is still desirable, in the event of a business reason to prefer a less optimal path or an overloaded source. But again, since you control the client, you can have it respond to your "chill out" packets and move to its next nearest target. That Sony doesn't appear to do any of this probably speaks more to a lack of expertise than anything else. They run a pretty decent online service with PSN, but I get the sense that they aren't really hiring people with a ton of Internet content distribution experience.

jsnell 9y ago
Sony control the client, but not the servers. They don't run their own CDN, but use at least half a dozen different commodity CDNs. So extra server side logic won't really be an option, it's just static files over HTTP in bulk. But yes, there's a lot of client side smarts they could benefit from.