4 ms·

(I work on Route 53). As __lucas has mentioned, this can be achieved with Route 53 Failover, which we'd recommend. Route 53 Failover is (hopefully) pretty eas

(I work on Route 53). As __lucas has mentioned, this can be achieved with Route 53 Failover, which we'd recommend. Route 53 Failover is (hopefully) pretty easy to configure. Just mark your ELB as the primary and enable target healthchecks, and add the S3 website bucket as the secondary. We'd also suggest that you use an S3 website bucket hosted in a different region than your ELB. This should take no more than a minute or two of setup in our console. The difference is makes is that Route 53 failover doesn't depend on any "make this change" control plane ; the status of every Route 53 healthcheck is constantly being polled from at at least 16 locations and then sent (via 3 redundant networking paths) to every Route 53 DNS server. The system is always operating at full-load, with a tremendous degree of fault-tolerance and redundancy. We hope that makes it very robust against very "hard to fix" internet and power problems, and also API outages. So for an awkward "worst case" example; if there were a large networking outage in an intermediate transit provider your customers might not be able to reach your ELB, and likewise you may not be able to reach the API to make the changes necessary. Route 53 failover should work anyway by detecting the reachability problem and flipping to the pre-configured secondary - an action which is triggered at our edge sites. If you'd rather not use Route 53 as your primary DNS provider that's ok; all of the above can still be achieved by using a dedicated zone on Route 53 just for managing the failover, which you may then CNAME to, just as with ELB. Each zone costs $0.50/month. Of course we'd also like to make this kind of functionality easier to use and built-in, and that's something we're constantly working on.

mnutt 13y ago
Do you guys have any plans to actually show the results of the DNS changes based on health checks? Connecting the health checks to cloudwatch is a great start, but it would be really useful to actually see which of my rules are in effect at any given time.
- colmmacc 13y ago
  Almost all of the time what you see in CloudWatch should be representative of what's being served. The internet stays consistent to at least four nines. Our CloudWatch metrics are actually made up of 16 independent metrics reporters (2 per 8 regions we run in) - each reporting a 1 or a 0. So you can already use the CloudWatch "average" statistic to get some limited visibility. Here's the metric for my own personal micro instance (hosted in us-west-2) ; http://failfast.info/micrometric.png http://failfast.info/micrometric.png . When the average drops to 0.875, what's actually happening is that one region couldn't reach us-west-2. (It turned out to be a transit networking issue somewhere between Singapore and Oregon). Just to give a sense of how rare partitions are; that's the first and only time I've seen one affect my instance, and I've been using the feature before we launched publicly. But yes, we would like to add visibility into things like where it seemed to fail from and whether it actually impacted any DNS decisions (in this case it didn't, as even our nodes in Singapore were able to tell that my endpoint was healthy, via redundant data from other regions). Edit to add: I should add that the most common reason why a health check fails is that the endpoint is just unhealthy, or totally unreachable. There is absolutely no ambiguity in those situations and the CloudWatch metrics tell you exactly when the failure and recovery happen. Only crazy internet partitions are hard.
  - aleem 13y ago
    Not directly related but I have one major gripe with ELB--am hopeful you have some inside knowledge and a possible workaround. I use CloudFlare for DDoS protection which Amazon doesn't offer. For apex domains, ELB requires route53 exclusively but that conflicts with CloudFlare which also requires DNS be hosted there. Since ELB uses CNAME records I haven't been able to find a way for the CloudFlare and ELB to work together (short of polling ELB IP changes every minute which doesn't really work out that well).
    - verelo 13y ago
      You can still use Route53 with Cloudflare by using their cname setup. This doesnt solve the issue for the apex however, so we have a S3 redirect on the apex. The key thing this allows us to do is continue to use Route53 with Cloudflare.
jedberg 13y ago
Except that his way all the traffic instantly switches, and your way you have to wait for DNS propagation, which about 15% of the users on the internet will not pick up for over a week. DNS is an awful way to do failover.
- colmmacc 13y ago
  To your point ; it's ok to do both. Route 53 supports DNS TTLs as low as 0 seconds. ELB and S3 endpoints both have 60 second TTLs. My experience with flipping names like www.amazon.com doesn't reflect the 15% figure. I've seen about 97% of web traffic honouring the TTL and flipping quickly. Within 5 minutes almost all of the rest too. We also take CloudFront sites in and out of service for maintenance, and in 5 years I've never anything like a 15% straggler effect. That said, we do see a very small number of stragglers. While resolvers over-riding TTLs hasn't shown up as a significant problem, buggy clients can be; we come across clients now and then who either don't re-resolve ever (Various JVMs and their infinite caches are a common cause), or only re-resolve on failures (which is fine for failover, but not great for traffic management). If you have a distribution time plot for the 15% figure it'd be interesting to see; https://www.dns-oarc.net/ https://www.dns-oarc.net/ would be a good venue, https://lists.dns-oarc.net/mailman/listinfo/dns-operations https://lists.dns-oarc.net/mailman/listinfo/dns-operations is the open list. Ignoring TTLs for a week is very concerning; it would very likely break many DNSSEC configurations. Is it possible you were dealing with robots?
  - btilly 13y ago
    I ran across the "DNS broken for a week", and it was broken for ISPs that run their own DNS for home customers. They didn't want to keep asking for upstream DNS, so just ignored the ttl. But I ran across this around the year 2000. And the biggest offender was named AOL. I could well believe that it is very, very different today.
  - ensignavenger 13y ago
    I don't know that I have ever seen a straggling DNS update take a week, but I have seen many take 3-4 days. Particularly Mediacom customers. I think this situation has improved some, but I still would not consider DNS failover very reliable.
ra 13y ago
Thanks this is a really great idea. I'm going to try this out.
VexXtreme 13y ago
We use weighted A records for this purpose. We basically have two weighted A records, one pointing to the load balancer and the other one to the static maintenance site on S3. When we want to switch sites, we just flip their weights (they are always 1 and 0), hence enforcing 100% redirection. It's not a perfect system but it works and DNS propagation is usually done within 15 minutes (haven't tried modifying the TTLs). Haven't been able to come up with anything simpler than this.
tszming 13y ago
The current R53 health check does not support SSL endpoints, which is rather critical for our use-cases. Please make it happen, thanks.
- revertts 13y ago
  Depending on your use case, you might be able to get away with a TCP check on 443 or a separate, HTTP accessible deep healthcheck.

mnutt 13y ago

Do you guys have any plans to actually show the results of the DNS changes based on health checks? Connecting the health checks to cloudwatch is a great start, but it would be really useful to actually see which of my rules are in effect at any given time.

colmmacc 13y ago
Almost all of the time what you see in CloudWatch should be representative of what's being served. The internet stays consistent to at least four nines. Our CloudWatch metrics are actually made up of 16 independent metrics reporters (2 per 8 regions we run in) - each reporting a 1 or a 0. So you can already use the CloudWatch "average" statistic to get some limited visibility. Here's the metric for my own personal micro instance (hosted in us-west-2) ; http://failfast.info/micrometric.png http://failfast.info/micrometric.png . When the average drops to 0.875, what's actually happening is that one region couldn't reach us-west-2. (It turned out to be a transit networking issue somewhere between Singapore and Oregon). Just to give a sense of how rare partitions are; that's the first and only time I've seen one affect my instance, and I've been using the feature before we launched publicly. But yes, we would like to add visibility into things like where it seemed to fail from and whether it actually impacted any DNS decisions (in this case it didn't, as even our nodes in Singapore were able to tell that my endpoint was healthy, via redundant data from other regions). Edit to add: I should add that the most common reason why a health check fails is that the endpoint is just unhealthy, or totally unreachable. There is absolutely no ambiguity in those situations and the CloudWatch metrics tell you exactly when the failure and recovery happen. Only crazy internet partitions are hard.
- aleem 13y ago
  Not directly related but I have one major gripe with ELB--am hopeful you have some inside knowledge and a possible workaround. I use CloudFlare for DDoS protection which Amazon doesn't offer. For apex domains, ELB requires route53 exclusively but that conflicts with CloudFlare which also requires DNS be hosted there. Since ELB uses CNAME records I haven't been able to find a way for the CloudFlare and ELB to work together (short of polling ELB IP changes every minute which doesn't really work out that well).
  - verelo 13y ago
    You can still use Route53 with Cloudflare by using their cname setup. This doesnt solve the issue for the apex however, so we have a S3 redirect on the apex. The key thing this allows us to do is continue to use Route53 with Cloudflare.

jedberg 13y ago

Except that his way all the traffic instantly switches, and your way you have to wait for DNS propagation, which about 15% of the users on the internet will not pick up for over a week. DNS is an awful way to do failover.

colmmacc 13y ago
To your point ; it's ok to do both. Route 53 supports DNS TTLs as low as 0 seconds. ELB and S3 endpoints both have 60 second TTLs. My experience with flipping names like www.amazon.com doesn't reflect the 15% figure. I've seen about 97% of web traffic honouring the TTL and flipping quickly. Within 5 minutes almost all of the rest too. We also take CloudFront sites in and out of service for maintenance, and in 5 years I've never anything like a 15% straggler effect. That said, we do see a very small number of stragglers. While resolvers over-riding TTLs hasn't shown up as a significant problem, buggy clients can be; we come across clients now and then who either don't re-resolve ever (Various JVMs and their infinite caches are a common cause), or only re-resolve on failures (which is fine for failover, but not great for traffic management). If you have a distribution time plot for the 15% figure it'd be interesting to see; https://www.dns-oarc.net/ https://www.dns-oarc.net/ would be a good venue, https://lists.dns-oarc.net/mailman/listinfo/dns-operations https://lists.dns-oarc.net/mailman/listinfo/dns-operations is the open list. Ignoring TTLs for a week is very concerning; it would very likely break many DNSSEC configurations. Is it possible you were dealing with robots?
- btilly 13y ago
  I ran across the "DNS broken for a week", and it was broken for ISPs that run their own DNS for home customers. They didn't want to keep asking for upstream DNS, so just ignored the ttl. But I ran across this around the year 2000. And the biggest offender was named AOL. I could well believe that it is very, very different today.
- ensignavenger 13y ago
  I don't know that I have ever seen a straggling DNS update take a week, but I have seen many take 3-4 days. Particularly Mediacom customers. I think this situation has improved some, but I still would not consider DNS failover very reliable.

ra 13y ago

Thanks this is a really great idea. I'm going to try this out.

VexXtreme 13y ago

We use weighted A records for this purpose. We basically have two weighted A records, one pointing to the load balancer and the other one to the static maintenance site on S3. When we want to switch sites, we just flip their weights (they are always 1 and 0), hence enforcing 100% redirection. It's not a perfect system but it works and DNS propagation is usually done within 15 minutes (haven't tried modifying the TTLs). Haven't been able to come up with anything simpler than this.

tszming 13y ago

The current R53 health check does not support SSL endpoints, which is rather critical for our use-cases. Please make it happen, thanks.

revertts 13y ago
Depending on your use case, you might be able to get away with a TCP check on 443 or a separate, HTTP accessible deep healthcheck.