5 ms·

It hadn't even occurred to me that someone might have considerable control/ownership over large tracts of DNS servers. (Of course they do, I'd just never though

It hadn't even occurred to me that someone might have considerable control/ownership over large tracts of DNS servers. (Of course they do, I'd just never thought about it!). So anyway, I'd just like to say thanks, this is a really great explanation.

walshemj 12y ago
I worked for a registry a few years ago and the uptime the ICANN mandate are very very tough full redundancy on 4 separate continents.
- toomuchtodo 12y ago
  Amazon, Google, Dnsmadeeasy, UltraDNS, Cloudflare all do it. I wouldn't say its that tough. Disclaimer: Infrastructure engineer
  - walshemj 12y ago
    Well maybe tough isn't the right word certainly expensive and unless your ex Telco and used to that level of redundancy a bit of a shock. And Amazon AWS remind me how many major down time incidents have they had ;-)
    - toomuchtodo 12y ago
      > And Amazon AWS remind me how many major down time incidents have they had ;-) Not sure if you were joking or not; I've managed thousands of instances in AWS across the world, and while us-east-1 continues to have issues occasionally, their DNS has been rock solid. Netflix uses them (with some UltraDNS for geolocation-based redirection, which Route53 doesn't support). That's good enough for me.
      - lsc 12y ago
        It's way easier to build a reliable DNS cluster than it is to build a reliable UNIX box. Anycast works way better for UDP/dns than it does for TCP/http. The idea behind Anycast is that you announce the address block your DNS cluster is on out many different locations around the world. At each location you have (usually a redundant cluster of) dns servers. The router ahead of the IP of the person looking up an address on your nameserver will check it's bgp table, and then choose whichever location of yours is the fewest hops away. From that router's point of view, this is no different from you having one location with many different service providers. If one of your locations goes away, it stops announcing the block, and all clients that would have gone to it instead use the next closest (hop-wise) location. (this is actually one of the hard parts; if you have a setup where the dns server stops responding correctly but it's still announcing via bgp, you are broken.) You generally don't use this for HTTP and other connection-based stuff, because a route flap is going to move your connection endpoint to an entirely new server. This causes a bunch of problems with your connection oriented stuff. (yes, you can sync connection state across servers... but the latency in that is going to kill you.)
        toomuchtodo 12y ago
        I've built global anycast DNS clusters before :) Thanks for sharing the info though! Protip: Perform service checks, and immediately withdraw your BGP route if a failure is detected. Also, props for explaining why HTTP anycast is a pain in the casee. Even stateless it doesn't work well, which is why its usually done with DNS vs BGP/anycast.
        lsc 12y ago
        >I've built global anycast DNS clusters before :) Hah. I've only talked about it. It's one of those things I want to do before it's an obsolete skill, but it isn't at the top of my priority list.
        toomuchtodo 12y ago
        You're not missing anything, trust me.

walshemj 12y ago

I worked for a registry a few years ago and the uptime the ICANN mandate are very very tough full redundancy on 4 separate continents.

toomuchtodo 12y ago
Amazon, Google, Dnsmadeeasy, UltraDNS, Cloudflare all do it. I wouldn't say its that tough. Disclaimer: Infrastructure engineer
- walshemj 12y ago
  Well maybe tough isn't the right word certainly expensive and unless your ex Telco and used to that level of redundancy a bit of a shock. And Amazon AWS remind me how many major down time incidents have they had ;-)
  - toomuchtodo 12y ago
    > And Amazon AWS remind me how many major down time incidents have they had ;-) Not sure if you were joking or not; I've managed thousands of instances in AWS across the world, and while us-east-1 continues to have issues occasionally, their DNS has been rock solid. Netflix uses them (with some UltraDNS for geolocation-based redirection, which Route53 doesn't support). That's good enough for me.
    - lsc 12y ago
      It's way easier to build a reliable DNS cluster than it is to build a reliable UNIX box. Anycast works way better for UDP/dns than it does for TCP/http. The idea behind Anycast is that you announce the address block your DNS cluster is on out many different locations around the world. At each location you have (usually a redundant cluster of) dns servers. The router ahead of the IP of the person looking up an address on your nameserver will check it's bgp table, and then choose whichever location of yours is the fewest hops away. From that router's point of view, this is no different from you having one location with many different service providers. If one of your locations goes away, it stops announcing the block, and all clients that would have gone to it instead use the next closest (hop-wise) location. (this is actually one of the hard parts; if you have a setup where the dns server stops responding correctly but it's still announcing via bgp, you are broken.) You generally don't use this for HTTP and other connection-based stuff, because a route flap is going to move your connection endpoint to an entirely new server. This causes a bunch of problems with your connection oriented stuff. (yes, you can sync connection state across servers... but the latency in that is going to kill you.)
      - toomuchtodo 12y ago
        I've built global anycast DNS clusters before :) Thanks for sharing the info though! Protip: Perform service checks, and immediately withdraw your BGP route if a failure is detected. Also, props for explaining why HTTP anycast is a pain in the casee. Even stateless it doesn't work well, which is why its usually done with DNS vs BGP/anycast.
        lsc 12y ago
        >I've built global anycast DNS clusters before :) Hah. I've only talked about it. It's one of those things I want to do before it's an obsolete skill, but it isn't at the top of my priority list.
        toomuchtodo 12y ago
        You're not missing anything, trust me.