4 ms·

Networking is so fragile. I feel like DNS failover to another AS in another datacenter is the only solution for web services to actually have resilience against

• toomuchtodo 10y ago
  Ops engineer here. You are correct.
  • diafygi 10y ago
    I'm a little confused about the mechanics of how dns failover works. Would you care to explain a bit more?
    • spotman 10y ago
      There is a few things possible here. You can do global load balancing which will serve a DNS entry for a hostname based on business rules. Those business rules can contain things like geo targeting, and/or they can also contain health checks. So you could direct traffic via dns based on if a region is online at all, or more granular things like is it meeting performance criteria, etc. Even more simple, is DNS round robin (just putting more than one entry in the reply for a dns request). So in terms of failover, if one of these servers in the reply stops working, generally another one is, and browsers and other devices generally understand that they can try all addresses listed for a given host.
      • deleted 10y ago
        [deleted]
      • ben_jones 10y ago
        Isn't DNS cached at some level on the users machines such that if a particular hostname went down clients which cached that hostname will still be directed to it? FWIW I'm making it a thing today to read up on DNS as this post made me woefully aware of how little I understand it. So feel free to skip this question and I'll probably answer it myself tonight.
        • toomuchtodo 10y ago
          Typically, with round robin DNS, the browser will try the other records if one fails. Its not perfect, but its good enough. In a perfect world, you use short TTLs (30-60 seconds), and withdraw a record from being advertised as soon as your health checks are failing for it.
          • ben_jones 10y ago
            That makes me think of something like an API-hungry mobile application that might make ~5 requests a second. If it had a 60-second cached DNS record to a host name that corresponded to a failed host that would lead to 300 failed requests and a really shitty user experience :(.
            • spotman 10y ago
              Your correct, but this is combatted in practice through UX usually. Doing work in the background, doing retry w/ decay, bubbling up things that didn't go well to the user at the right time, etc. For a client to just spam 300 requests the moment there is a hiccup that is generally not good. At scale its indistinguishable from a DDOS (been there, hah!). Also for it to be that jarring we are talking about a failure at the exact moment you are using the data-heavy app, and not 1 minute before. Every time you go to use the app its likely the dns is re-queried. So, at the end of the day if you have a mission critical thing running where you need to try very hard and can't tolerate 60 seconds of failure, you would bake multiple paths to a backend in the client configuration at some point. So the client would know if backend A is down it can do client side failover to backend B. Sometimes this technique is used in conjunction with a highly fault tolerant server stuff, and not as a replacement for it.
              • ben_jones 10y ago
                Thanks for the follow-up, it's always insightful to hear from people who have seen these things in the trenches on larger systems then I've had privy to.
• BinaryIdiot 10y ago
  I feel like DNS is one of the best ways to do service discovery but its complexity is always baffling to me. I mean I understand roughly how it all works but if you asked me to stand up DNS internally for various servers to use it...might take me a while. I've wondered if it's simply because I don't know it well enough or if we need an easier, more simple solution to replace DNS in internal networks. There are a ton of ways to do service discovery.
  • eropple 10y ago
    > I've wondered if it's simply because I don't know it well enough IMO, and emphatically not ripping on you: it's this one. DNS is a pretty simple protocol and at "you don't have ten thousand developers" scale managing it is pretty easy. It might take you a while the first time, for sure--but it won't the second, third, or twelfth time. =) In particular, even setting aside explicit failover cases (which do require some outside monitoring and staging to make useful, i.e. when do you failover?), round-robin DNS actually does a lot to provide redundancy, internal to a single data center or deployment, when coupled with SRV records.
    • BinaryIdiot 10y ago
      Fair enough. Any suggested reading material regarding this subject?
      • thenewwazoo 10y ago
        I honestly recommend just setting up a toy split-view DNS service with a hidden master using BIND9. Get zone transfers working and maybe play with dynamic updates, and you'll basically know all you need to know. I set up an internal DNS infra at a previous job and was pretty surprised at how simple it really ended up being.
        • BinaryIdiot 10y ago
          Thanks! I'll take a look.
• falcolas 10y ago
  If only DNS failover were as atomic as would be required to do good failover. Sadly, so many clients and caching mechanisms ignore TTL values that an actual failover (not to mention fail back) could take days.
• lazyant 10y ago
  It's "a" solution but not perfect since now you depend on the clients to use the new records (caches, respecting TTL), there's also propagation times etc.
  • zzzcpan 10y ago
    What are the alternatives? I don't think you can prevent problems affecting the whole AS or problems affecting the whole company with anything but failing over to another AS or another company. And the only place you can do that for a web browser is DNS. And there is some beauty in it. It's fairly simple and with more servers in more places you get fewer affecting clients when one of the places fails and fewer disrespectful clients as well. It scales very well and cheaply.