3 ms·

Their comment implies "are totally fine with stale data". Their argument is that the membership set for a service (especially on-prem) doesn't change all that

by boulos 5y ago

Their comment implies "are totally fine with stale data". Their argument is that the membership set for a service (especially on-prem) doesn't change all that frequently, and even if it's out of date, it's likely that most of the endpoints are still actually servicing the thing you were looking for. That plus client retries and you're often pretty good.

tptacek 5y ago
Maybe I'm just working on an idiosyncratic version of the service discovery problem, but "stale data" is basically my bête noire. Part of it is that I don't control all my clients, and I can't guarantee they have sane retry logic; what service discovery tells them is the best place to go had better be responsive, or we're effectively having an outage. For us, service discovery is exquisitely sensitive to stale data.
- boulos 5y ago
  Yep! I'm not saying I totally agree with the original comment there, just confirming that they meant it. If you own your clients, sometimes you can say "it's on you to retry" (deadlines and retries are effectively mandatory, and often automatic, at Google). Having user facing services hand out bad addresses / endpoints would be really bad. However, even for things like databases, you really want to know who the leader / primary is (and it's not really okay to get a stale answer). So I dunno, some things are just fine with it, and some aren't. It's better if it just works :). Besides, if the data isn't changing, the write rate isn't high!
  - NightMKoder 5y ago
    Yes - exactly what boulos said - I’m coming from the Google “you control your clients” perspective. That said, in some sense you always control your clients - you can always set up localhost proxies that speak your protocol or just tcp proxies. The thing is service discovery is _always_ inconsistent. By the time you get your set of endpoints from your discovery, it can be out of date by the time you open your socket. Certainly for something like databases you need followers or standby leaders to reject writes from clients - service discovery can’t 100% save you here.

by boulos 5y ago

tptacek 5y ago

Maybe I'm just working on an idiosyncratic version of the service discovery problem, but "stale data" is basically my bête noire. Part of it is that I don't control all my clients, and I can't guarantee they have sane retry logic; what service discovery tells them is the best place to go had better be responsive, or we're effectively having an outage. For us, service discovery is exquisitely sensitive to stale data.

boulos 5y ago
Yep! I'm not saying I totally agree with the original comment there, just confirming that they meant it. If you own your clients, sometimes you can say "it's on you to retry" (deadlines and retries are effectively mandatory, and often automatic, at Google). Having user facing services hand out bad addresses / endpoints would be really bad. However, even for things like databases, you really want to know who the leader / primary is (and it's not really okay to get a stale answer). So I dunno, some things are just fine with it, and some aren't. It's better if it just works :). Besides, if the data isn't changing, the write rate isn't high!
- NightMKoder 5y ago
  Yes - exactly what boulos said - I’m coming from the Google “you control your clients” perspective. That said, in some sense you always control your clients - you can always set up localhost proxies that speak your protocol or just tcp proxies. The thing is service discovery is _always_ inconsistent. By the time you get your set of endpoints from your discovery, it can be out of date by the time you open your socket. Certainly for something like databases you need followers or standby leaders to reject writes from clients - service discovery can’t 100% save you here.