3 ms·

I'm excited for all the buzz going on around service discovery, it's often one of the most crucial design decisions for a SOA architecture. I'm really bummed, h

by jolynch 11y ago

I'm excited for all the buzz going on around service discovery, it's often one of the most crucial design decisions for a SOA architecture. I'm really bummed, however, that so many of them rely on clients doing DNS lookups. I vastly prefer a system like airbnb/Yelp SmartStack, consul template, or really any system that reflects discovery state in real load balancers such as HAProxy, nginx, or even those new fangled service load balancers like vulcand. Honestly just don't do DNS. Why? 1. Performance Service discovery should not happen during the life of a HTTP request. The number of times that I've had to debug performance problem that ultimately ended up being DNS requests taking like 100ms (which for most internal applications is too damn long) is staggering to me. Yes you can mitigate this, but it is not easy because even local DNS has to miss the cache sometimes. 2. DNS propagation sucks. This article talks about one side of the DNS propagation problem ("it's down" -> removed from DNS), but they seem to completely ignore the other side of the DNS propagation problem (removed from dns -> clients stop talking to it). At any kind of reasonable engineering scale one of your libraries (watch out for ones that cache DNS "for performance"), programming languages (java), or infra tools (nginx/haproxy) are going to not re-resolve DNS like you expect them to. 3. More Performance Client side load balancing is legitimately hard, and if you're just doing round robin over DNS records you are likely screwing your high percentiles. HAProxy/nginx/vulcand/etc all support better load balancing strategies like least connection load balancing that really make a difference here. This can be the difference between a 500ms 99% and a 50ms 99%.

fastest963 11y ago
These are very good points. I'll be honest, I don't have a lot of experience with SmartStack, but I'll be reviewing it. As far as caching, ideally you'd have no caching, however, certain optimizations can be made on this front, such as combining in-flight-and-duplicate queries. Load balancing could be done via the SRV records, but that would be difficult since it requires that you republish the SRV record. This could be done every minute or so based on load but not on a per-request basis. Thanks for your insightful comment!
KaiserPro 11y ago
on point 1, I've not seen local authoritative DNS take that long to resolve a local address. If the local DNS has to recurse out to find an address, then yes. But if thats the case, then something has gone wrong with the setup. (this is local service discovery after all.) on point 2: Yes, long running processes cache DNS. so does SSSD and nslcd. However in the world of containers, the first point of call for debugging appears to redeploy, so this goes away, by accident. (and not entirely convincingly) on point 3, yes, yes very much.

by jolynch 11y ago

fastest963 11y ago

These are very good points. I'll be honest, I don't have a lot of experience with SmartStack, but I'll be reviewing it. As far as caching, ideally you'd have no caching, however, certain optimizations can be made on this front, such as combining in-flight-and-duplicate queries. Load balancing could be done via the SRV records, but that would be difficult since it requires that you republish the SRV record. This could be done every minute or so based on load but not on a per-request basis. Thanks for your insightful comment!

KaiserPro 11y ago

on point 1, I've not seen local authoritative DNS take that long to resolve a local address. If the local DNS has to recurse out to find an address, then yes. But if thats the case, then something has gone wrong with the setup. (this is local service discovery after all.) on point 2: Yes, long running processes cache DNS. so does SSSD and nslcd. However in the world of containers, the first point of call for debugging appears to redeploy, so this goes away, by accident. (and not entirely convincingly) on point 3, yes, yes very much.