5 ms·

But there must be something before ≈ nginx as well right? Something something BGP to route it to the closest POP? And when you have traffic to a single pop that

But there must be something before ≈ nginx as well right? Something something BGP to route it to the closest POP? And when you have traffic to a single pop that are multiples of what even the beefiest machine can handle, something before the machine must load balance it I guess?

efitz 4y ago
That’s done by DNS. The DNS server examines your DNS query and sends you back an IP address answer that’s “close” to the requesting IP address. For example, if you are in country X, then even though there are dozens or hundreds of POP addresses for the name you requested, you’ll be answered with one for country X.
- nik736 4y ago
  This can also be done with BGP. Anycast BGP for example is way better suited for this use-case.
  - cryptonym 4y ago
    BGP is an alternative to DNS but it is not better suited, it depends. DNS gives more control and precision to CDN provider than BGP. BGP might be good enough if you got few POP around the world but with the scale/distribution of Akamai, DNS is better suited.
    - ThePhysicist 4y ago
      I think all major CDNs use some form of anycasting as it's pretty essential that you own your IP space. DNS-based load balancing can also be finicky as you'll have to deal with recursive resolvers, so a combination of anycasting and DNS-based load balancing probably works best.
    - manigandham 4y ago
      How does DNS give you more control than BGP? One is an actual routing system that tells routers where to send data, the other is a name translation system with multiple layers of caches outside of your control. DNS is a layer above as BGP is still used to actually navigate to the listed IP, and any large CDN will own that IP space and announce their own routes anyway.
      - capableweb 4y ago
        > DNS is a layer above as BGP is still used to actually navigate to the listed IP, and any large CDN will own that IP space and announce their own routes anyway. Yeah, I'm not sure what parent is on about, BGP and DNS are not alternatives to each other, the internet relies on both of them but at different layers. Without BGP packets wouldn't know how to be routed and without DNS they wouldn't know where to be routing to, they are complementary.
        cryptonym 4y ago
        Sure, Internet communications relies on both and you cannot swap one with another. This thread is about load balancing and both can be used for load balancing (Basically DNS resolving to Anycast IP vs DNS resolving to Unicast IPs).
        sophacles 4y ago
        What's the difference between an anycast IP and a unicast IP? Answer: Nothing. They are just IPs. Using a combination of Anycast and DNS is going to give the best control over steering http traffic. Particularly if you own a few prefixes and can do clever addressing tricks.
      - daper 4y ago
        With BGP (especially anycast) you don't have direct influence where the requests lands. You can steer traffic by techniques like AS prepend, some priorities per BGP session (I'm not an expert in BGP) but eventually it's not CDN that decides where the request will be routed. It's decided by routers of the client's ISP and backbone networks, each taking decision themself (shorted AS path from his point of view + BGP policies set by network operators), not by you. You can't for example split traffic between DCs with specific proportion (like 30% here and 70% there). You can't split by something else than network properties like forward request to the region that has better chance to have this content in the cache. With DNS and dynamic responses you are directing request to specific DC, even server, almost on every request. It may be dedicated for this traffic type (live stream different than static images etc). Your DNS server can take the hostname ("www.google.com") into consideration - BGP doesn't even know the hostname in the URL. If you wanted to do it with BGP you would need to place specific content to a dedicated /24 subnet and that is impossible considering how many IPv4 addresses are available. BGP doesn't even consider network latency, current network load. CDN knows load on their machines, on their network link, where given content is placed. The bottleneck may be storage, network or CPU processing, different for different sites and content type. They need to direct traffic on request basis considering this and at least the hostname from the URL. That's why DNS is used first.
  - wbl 4y ago
    Anycast doesn't let you adjust traffic flows gradually to shed load: you end up breaking connections. DNS is more flexible but isn't always honored.
- capableweb 4y ago
  > That’s done by DNS. The DNS server examines your DNS query and sends you back an IP address answer that’s “close” to the requesting IP address. Is this really true? Last time I was dealing with load-balancing and DNS queries, DNS was simply "round-robin" the replies, giving you back a random record basically of the ones replied. So if you have three A records with different IPs, each query will give you back one of them, but not depending on the location. Maybe things have changed since I last dealt with it, but the DNS ecosystem doesn't tend to move very fast so I'm doubtful...
  - pjc50 4y ago
    The DNS server doesn't have to return the same responses to every query. It can geolocate the address making the request and use that to determine the respone. https://easydns.com/features/geo-dns/ https://easydns.com/features/geo-dns/ (Having said that, I tried my toy site on cloudflare free tier from the UK and it gave me San Francisco IPs, so presumably they only do this for large enough customers)
    - capableweb 4y ago
      > It can geolocate the address making the request and use that to determine the respone It can, but it tends to be a premium feature of specific DNS providers, not a global/by-default feature of DNS as efitz seems to be alluding to. DNSimple supports it for example, but only on their "Professional" plan (and they call it "Regional Records") while others like Gandi don't support it at all.
      - strken 4y ago
        If CDN providers are dealing with BGP you'd think they would also run their own DNS servers. I think you can do it with BIND views, from memory.
      - deleted 4y ago
        [deleted]
    - jonathantf2 4y ago
alfons_foobar 4y ago
BGP Anycast (possibly in combination with DNS depending on geolocation). Basically you announce the same IP address from multiple locations, and BGP chooses the best* Route to reach that address. * Note that "best" does not nessecarily mean "fastest", not even "least hops" - BGP has many knobs where the routing decisions can be manipulated, but by default it choses the route where traffic has to traverse the smallest number of "Autonomous Systems" (roughly equivalent to "Organizational Entities").

efitz 4y ago

That’s done by DNS. The DNS server examines your DNS query and sends you back an IP address answer that’s “close” to the requesting IP address. For example, if you are in country X, then even though there are dozens or hundreds of POP addresses for the name you requested, you’ll be answered with one for country X.

nik736 4y ago
This can also be done with BGP. Anycast BGP for example is way better suited for this use-case.
- cryptonym 4y ago
  BGP is an alternative to DNS but it is not better suited, it depends. DNS gives more control and precision to CDN provider than BGP. BGP might be good enough if you got few POP around the world but with the scale/distribution of Akamai, DNS is better suited.
  - ThePhysicist 4y ago
    I think all major CDNs use some form of anycasting as it's pretty essential that you own your IP space. DNS-based load balancing can also be finicky as you'll have to deal with recursive resolvers, so a combination of anycasting and DNS-based load balancing probably works best.
  - manigandham 4y ago
    How does DNS give you more control than BGP? One is an actual routing system that tells routers where to send data, the other is a name translation system with multiple layers of caches outside of your control. DNS is a layer above as BGP is still used to actually navigate to the listed IP, and any large CDN will own that IP space and announce their own routes anyway.
    - capableweb 4y ago
      > DNS is a layer above as BGP is still used to actually navigate to the listed IP, and any large CDN will own that IP space and announce their own routes anyway. Yeah, I'm not sure what parent is on about, BGP and DNS are not alternatives to each other, the internet relies on both of them but at different layers. Without BGP packets wouldn't know how to be routed and without DNS they wouldn't know where to be routing to, they are complementary.
      - cryptonym 4y ago
        Sure, Internet communications relies on both and you cannot swap one with another. This thread is about load balancing and both can be used for load balancing (Basically DNS resolving to Anycast IP vs DNS resolving to Unicast IPs).
        sophacles 4y ago
        What's the difference between an anycast IP and a unicast IP? Answer: Nothing. They are just IPs. Using a combination of Anycast and DNS is going to give the best control over steering http traffic. Particularly if you own a few prefixes and can do clever addressing tricks.
    - daper 4y ago
      With BGP (especially anycast) you don't have direct influence where the requests lands. You can steer traffic by techniques like AS prepend, some priorities per BGP session (I'm not an expert in BGP) but eventually it's not CDN that decides where the request will be routed. It's decided by routers of the client's ISP and backbone networks, each taking decision themself (shorted AS path from his point of view + BGP policies set by network operators), not by you. You can't for example split traffic between DCs with specific proportion (like 30% here and 70% there). You can't split by something else than network properties like forward request to the region that has better chance to have this content in the cache. With DNS and dynamic responses you are directing request to specific DC, even server, almost on every request. It may be dedicated for this traffic type (live stream different than static images etc). Your DNS server can take the hostname ("www.google.com") into consideration - BGP doesn't even know the hostname in the URL. If you wanted to do it with BGP you would need to place specific content to a dedicated /24 subnet and that is impossible considering how many IPv4 addresses are available. BGP doesn't even consider network latency, current network load. CDN knows load on their machines, on their network link, where given content is placed. The bottleneck may be storage, network or CPU processing, different for different sites and content type. They need to direct traffic on request basis considering this and at least the hostname from the URL. That's why DNS is used first.
- wbl 4y ago
  Anycast doesn't let you adjust traffic flows gradually to shed load: you end up breaking connections. DNS is more flexible but isn't always honored.
capableweb 4y ago
> That’s done by DNS. The DNS server examines your DNS query and sends you back an IP address answer that’s “close” to the requesting IP address. Is this really true? Last time I was dealing with load-balancing and DNS queries, DNS was simply "round-robin" the replies, giving you back a random record basically of the ones replied. So if you have three A records with different IPs, each query will give you back one of them, but not depending on the location. Maybe things have changed since I last dealt with it, but the DNS ecosystem doesn't tend to move very fast so I'm doubtful...
- pjc50 4y ago
  The DNS server doesn't have to return the same responses to every query. It can geolocate the address making the request and use that to determine the respone. https://easydns.com/features/geo-dns/ https://easydns.com/features/geo-dns/ (Having said that, I tried my toy site on cloudflare free tier from the UK and it gave me San Francisco IPs, so presumably they only do this for large enough customers)
  - capableweb 4y ago
    > It can geolocate the address making the request and use that to determine the respone It can, but it tends to be a premium feature of specific DNS providers, not a global/by-default feature of DNS as efitz seems to be alluding to. DNSimple supports it for example, but only on their "Professional" plan (and they call it "Regional Records") while others like Gandi don't support it at all.
    - strken 4y ago
      If CDN providers are dealing with BGP you'd think they would also run their own DNS servers. I think you can do it with BIND views, from memory.
    - deleted 4y ago
      [deleted]
  - jonathantf2 4y ago

alfons_foobar 4y ago

BGP Anycast (possibly in combination with DNS depending on geolocation). Basically you announce the same IP address from multiple locations, and BGP chooses the best* Route to reach that address. * Note that "best" does not nessecarily mean "fastest", not even "least hops" - BGP has many knobs where the routing decisions can be manipulated, but by default it choses the route where traffic has to traverse the smallest number of "Autonomous Systems" (roughly equivalent to "Organizational Entities").