4 ms·

Isn't DNS cached at some level on the users machines such that if a particular hostname went down clients which cached that hostname will still be directed to i

Isn't DNS cached at some level on the users machines such that if a particular hostname went down clients which cached that hostname will still be directed to it? FWIW I'm making it a thing today to read up on DNS as this post made me woefully aware of how little I understand it. So feel free to skip this question and I'll probably answer it myself tonight.

toomuchtodo 10y ago
Typically, with round robin DNS, the browser will try the other records if one fails. Its not perfect, but its good enough. In a perfect world, you use short TTLs (30-60 seconds), and withdraw a record from being advertised as soon as your health checks are failing for it.
- ben_jones 10y ago
  That makes me think of something like an API-hungry mobile application that might make ~5 requests a second. If it had a 60-second cached DNS record to a host name that corresponded to a failed host that would lead to 300 failed requests and a really shitty user experience :(.
  - spotman 10y ago
    Your correct, but this is combatted in practice through UX usually. Doing work in the background, doing retry w/ decay, bubbling up things that didn't go well to the user at the right time, etc. For a client to just spam 300 requests the moment there is a hiccup that is generally not good. At scale its indistinguishable from a DDOS (been there, hah!). Also for it to be that jarring we are talking about a failure at the exact moment you are using the data-heavy app, and not 1 minute before. Every time you go to use the app its likely the dns is re-queried. So, at the end of the day if you have a mission critical thing running where you need to try very hard and can't tolerate 60 seconds of failure, you would bake multiple paths to a backend in the client configuration at some point. So the client would know if backend A is down it can do client side failover to backend B. Sometimes this technique is used in conjunction with a highly fault tolerant server stuff, and not as a replacement for it.
    - ben_jones 10y ago
      Thanks for the follow-up, it's always insightful to hear from people who have seen these things in the trenches on larger systems then I've had privy to.

toomuchtodo 10y ago

Typically, with round robin DNS, the browser will try the other records if one fails. Its not perfect, but its good enough. In a perfect world, you use short TTLs (30-60 seconds), and withdraw a record from being advertised as soon as your health checks are failing for it.

ben_jones 10y ago
That makes me think of something like an API-hungry mobile application that might make ~5 requests a second. If it had a 60-second cached DNS record to a host name that corresponded to a failed host that would lead to 300 failed requests and a really shitty user experience :(.
- spotman 10y ago
  Your correct, but this is combatted in practice through UX usually. Doing work in the background, doing retry w/ decay, bubbling up things that didn't go well to the user at the right time, etc. For a client to just spam 300 requests the moment there is a hiccup that is generally not good. At scale its indistinguishable from a DDOS (been there, hah!). Also for it to be that jarring we are talking about a failure at the exact moment you are using the data-heavy app, and not 1 minute before. Every time you go to use the app its likely the dns is re-queried. So, at the end of the day if you have a mission critical thing running where you need to try very hard and can't tolerate 60 seconds of failure, you would bake multiple paths to a backend in the client configuration at some point. So the client would know if backend A is down it can do client side failover to backend B. Sometimes this technique is used in conjunction with a highly fault tolerant server stuff, and not as a replacement for it.
  - ben_jones 10y ago
    Thanks for the follow-up, it's always insightful to hear from people who have seen these things in the trenches on larger systems then I've had privy to.