3 ms·

> If there is an issue with a collector, Route 53 automatically takes it out of the service; our customers won’t see any impact. Except when for example rsyslo

by zimbatm 12y ago

> If there is an issue with a collector, Route 53 automatically takes it out of the service; our customers won’t see any impact. Except when for example rsyslog caches DNS resolution forever. Or the log forwarded doesn't have a buffer and logs get lost.

general_failure 12y ago
How does this work regardless? Isn't route53 a DNS system? How does it know whether a service went up/down? EDIT: To reply to myself http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/health-checks-creating-deleting.html http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hea... http://docs.aws.amazon.com/Route53/latest/APIReference/API_CreateHealthCheck.html http://docs.aws.amazon.com/Route53/latest/APIReference/API_C...
- zimbatm 12y ago
  Unfortunately inside of AWS there is no other solution than to rely on DNS to update the list of IP that resolve from an address (even ELB does that). Route53 has health-checks that can update the list automatically. Uptime is entirely dependent on the TTL of those records and if the clients respects that TTL. Complementary to that it's also possible to assign reserved IPs to machines so that at least the set of IP is always the same even if the hosts get rotated. Assigning IPs to hosts is not instantaneous either and depends on an API call. Also IPs are tied to a specific region. In a real datacenter other options are possible like sharing IP addresses between multiple devices and having lower-latency failover. These aren't perfect either and have different failure scenarios.
korzun 12y ago
Yeah I don't get their approach. There is no way this will cause 100% delivery if one server fails within that rotation. That chances of failure go up dramatically if 2+ hosts behind round robin fail, etc. Not to mention once hosts resolve this to an IP they will re-use the route. This approach is not balanced. I don't want to be /that/ guy but if they can't scale with ELB they should invest in a dedicated load balancer infrastructure that can offload requests to their cloud instances. This is a really bizarre post.
- hatred 12y ago
  I am more worried about the DNS caching issue. The 2+ hosts failing should not be much of a problem if you have a separate health checker host which does nothing except gathering heart-beats from all the hosts in your fleet and updating the DNS periodically.
  - korzun 12y ago
    That's exactly why it's a problem. If you have 3 hosts and 2 of them go down, in this setup there is more than 50% chance that cached hosts will be trying to connect to a non-existing server. Also expecting client to perform a DNS lookup every time there is an outgoing log packet is pretty shitty for performance. You can't guarantee near instant DNS server availability for every client.
  - hijinks 12y ago
    not to mention people that use something like a local caching daemon like nscd might cache a bad ip for a whole day
- boulos 12y ago
  Honestly it sounds like they just need a high performance L3 load balancer, rather than ELB. As you said though, just because AWS doesn't give them that doesn't mean the slot up was Route53 instead of say haproxy.

by zimbatm 12y ago

general_failure 12y ago

How does this work regardless? Isn't route53 a DNS system? How does it know whether a service went up/down? EDIT: To reply to myself http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/health-checks-creating-deleting.html http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hea... http://docs.aws.amazon.com/Route53/latest/APIReference/API_CreateHealthCheck.html http://docs.aws.amazon.com/Route53/latest/APIReference/API_C...

zimbatm 12y ago
Unfortunately inside of AWS there is no other solution than to rely on DNS to update the list of IP that resolve from an address (even ELB does that). Route53 has health-checks that can update the list automatically. Uptime is entirely dependent on the TTL of those records and if the clients respects that TTL. Complementary to that it's also possible to assign reserved IPs to machines so that at least the set of IP is always the same even if the hosts get rotated. Assigning IPs to hosts is not instantaneous either and depends on an API call. Also IPs are tied to a specific region. In a real datacenter other options are possible like sharing IP addresses between multiple devices and having lower-latency failover. These aren't perfect either and have different failure scenarios.

korzun 12y ago

Yeah I don't get their approach. There is no way this will cause 100% delivery if one server fails within that rotation. That chances of failure go up dramatically if 2+ hosts behind round robin fail, etc. Not to mention once hosts resolve this to an IP they will re-use the route. This approach is not balanced. I don't want to be /that/ guy but if they can't scale with ELB they should invest in a dedicated load balancer infrastructure that can offload requests to their cloud instances. This is a really bizarre post.

hatred 12y ago
I am more worried about the DNS caching issue. The 2+ hosts failing should not be much of a problem if you have a separate health checker host which does nothing except gathering heart-beats from all the hosts in your fleet and updating the DNS periodically.
- korzun 12y ago
  That's exactly why it's a problem. If you have 3 hosts and 2 of them go down, in this setup there is more than 50% chance that cached hosts will be trying to connect to a non-existing server. Also expecting client to perform a DNS lookup every time there is an outgoing log packet is pretty shitty for performance. You can't guarantee near instant DNS server availability for every client.
- hijinks 12y ago
  not to mention people that use something like a local caching daemon like nscd might cache a bad ip for a whole day
boulos 12y ago
Honestly it sounds like they just need a high performance L3 load balancer, rather than ELB. As you said though, just because AWS doesn't give them that doesn't mean the slot up was Route53 instead of say haproxy.