8 ms·

Don't want to go off topic, but if I want to prevent my website going down because of stuff like this in the future, will having back up DNS entries solve the p

Don't want to go off topic, but if I want to prevent my website going down because of stuff like this in the future, will having back up DNS entries solve the problem? I know DNS will fall back if it can't reach a service, but would a 502 trigger that?

rosswilson 7y ago
If you received a HTTP 502 then DNS must've already resolved. Browsers typically will do a DNS lookup, and then try establishing a TCP connection to one of the returned hosts. Its only if it can't establish a TCP connection to a host will it (sometimes) try another host from the DNS response.
lol768 7y ago
No, I don't believe it would.
SteveNuts 7y ago
No, you'd need to use an application-aware DNS provider like Route 53 that could detect the failure. Backup DNS entries won't help, what you need is to use multiple DNS providers and add them all to your NS records
- danShumway 7y ago
  I think this is the correct answer. Since I'm mostly thinking about static sites, I could also have something local that pings the site, and if it goes down, it could update my nameservers to point elsewhere on its own. Probably more trouble than its worth though.
aclelland 7y ago
Yeah, you'll want to keep your domain registered through a different registrar and if CF goes down you can update your DNS Name Servers point from CF to something like AWS Route 53. This has a few drawbacks like making sure your Route53 configuration is identical to your CF config, ensuring your origin servers can cope with the additional load if CF caching isn't available and the DNS propagation time required for the Name Servers to update. During the last outage, we were able to get into the CF dashboard and simply disable the proxy which allowed our clients to access our origin servers directly but this time we can't even get into the Dashboard.
- danShumway 7y ago
  Yeah, if I had access to the DNS records this would be easy, but like you said, even the dashboard is down. Ideally, I'd want something where if Cloudflare goes down, I don't have to change anything, but... 502 isn't going to trigger that without some work on my part. Meh.
iofiiiiiiiii 7y ago
What would solve it is having your DNS have a low lifetime and then changing the DNS to point to not-Cloudflare. It would still be down for some users as long as the old (Cloudflare) DNS is cached, though.
- aeonflux 7y ago
  Make sure that DNS provider offers an API and that none of their infrastructure is hosted on CF. :D
aeonflux 7y ago
No, dns doesn't work like this, you can't have backup records.
- LeonM 7y ago
  Not necessarily backup records, but you can add multiple A/AAAA records. There is no guaranteed order though. It is possible to have multiple A/AAAA records pointing to different loadbalancers, but I don't know how browsers would deal with this.
  - aeonflux 7y ago
    The lookup is essentially random. If you point DNS to 2 IP's and one of those goes down, then (without going into detail) half of your requests will fail.
    - LeonM 7y ago
      Yes, that is correct. But it does make it possible for a browser to retry a different IP if a 500 range error, or timeout occurs. But like I said in my other post: I don't know if browsers actually do this.
  - LinuxBender 7y ago
    Browsers would deal with it just fine (assuming the site is down hard and not responding with errors). Its some of the API tools and old libraries that may not. They would need retry logic that mimics the browser cycling through multiple A records. OTOH, API tools that have retry logic would just keep trying until the errors clear up. A browser will stop retrying when something responds unless there was javascript running in memory that had retry logic. If there are errors, the site would need to be modified to not respond if broken and unable to proxy to a working origin. Perhaps CF have not coded their proxies in this manor.
LeonM 7y ago
This is a good question. I'm not sure if the browser receives all A/AAAA records from the syscall, or just one. I guess that if the browser has the whole list, and the error is in the 500 range, it could retry a different IP but I'm not sure if browsers do this.
- swiley 7y ago
  DNS isn’t handled by the kernel, it’s handled by the network library runtime and that does return a list of addresses (I think it actually has you iterate through them in C anyway.)
partiallypro 7y ago
A problem with this though is that some registrars take hours to propagate, by the time you have it switched it will have already likely been resolved. If you spread that across hundreds of customers, you'd have a bad time.

rosswilson 7y ago

If you received a HTTP 502 then DNS must've already resolved. Browsers typically will do a DNS lookup, and then try establishing a TCP connection to one of the returned hosts. Its only if it can't establish a TCP connection to a host will it (sometimes) try another host from the DNS response.

lol768 7y ago

No, I don't believe it would.

SteveNuts 7y ago

No, you'd need to use an application-aware DNS provider like Route 53 that could detect the failure. Backup DNS entries won't help, what you need is to use multiple DNS providers and add them all to your NS records

danShumway 7y ago
I think this is the correct answer. Since I'm mostly thinking about static sites, I could also have something local that pings the site, and if it goes down, it could update my nameservers to point elsewhere on its own. Probably more trouble than its worth though.

aclelland 7y ago

Yeah, you'll want to keep your domain registered through a different registrar and if CF goes down you can update your DNS Name Servers point from CF to something like AWS Route 53. This has a few drawbacks like making sure your Route53 configuration is identical to your CF config, ensuring your origin servers can cope with the additional load if CF caching isn't available and the DNS propagation time required for the Name Servers to update. During the last outage, we were able to get into the CF dashboard and simply disable the proxy which allowed our clients to access our origin servers directly but this time we can't even get into the Dashboard.

danShumway 7y ago
Yeah, if I had access to the DNS records this would be easy, but like you said, even the dashboard is down. Ideally, I'd want something where if Cloudflare goes down, I don't have to change anything, but... 502 isn't going to trigger that without some work on my part. Meh.

iofiiiiiiiii 7y ago

What would solve it is having your DNS have a low lifetime and then changing the DNS to point to not-Cloudflare. It would still be down for some users as long as the old (Cloudflare) DNS is cached, though.

aeonflux 7y ago
Make sure that DNS provider offers an API and that none of their infrastructure is hosted on CF. :D

aeonflux 7y ago

No, dns doesn't work like this, you can't have backup records.

LeonM 7y ago
Not necessarily backup records, but you can add multiple A/AAAA records. There is no guaranteed order though. It is possible to have multiple A/AAAA records pointing to different loadbalancers, but I don't know how browsers would deal with this.
- aeonflux 7y ago
  The lookup is essentially random. If you point DNS to 2 IP's and one of those goes down, then (without going into detail) half of your requests will fail.
  - LeonM 7y ago
    Yes, that is correct. But it does make it possible for a browser to retry a different IP if a 500 range error, or timeout occurs. But like I said in my other post: I don't know if browsers actually do this.
- LinuxBender 7y ago
  Browsers would deal with it just fine (assuming the site is down hard and not responding with errors). Its some of the API tools and old libraries that may not. They would need retry logic that mimics the browser cycling through multiple A records. OTOH, API tools that have retry logic would just keep trying until the errors clear up. A browser will stop retrying when something responds unless there was javascript running in memory that had retry logic. If there are errors, the site would need to be modified to not respond if broken and unable to proxy to a working origin. Perhaps CF have not coded their proxies in this manor.

LeonM 7y ago

This is a good question. I'm not sure if the browser receives all A/AAAA records from the syscall, or just one. I guess that if the browser has the whole list, and the error is in the 500 range, it could retry a different IP but I'm not sure if browsers do this.

swiley 7y ago
DNS isn’t handled by the kernel, it’s handled by the network library runtime and that does return a list of addresses (I think it actually has you iterate through them in C anyway.)

partiallypro 7y ago

A problem with this though is that some registrars take hours to propagate, by the time you have it switched it will have already likely been resolved. If you spread that across hundreds of customers, you'd have a bad time.