4 ms·
Somewhat related question: Consul's deployment guide recommends against deploying a cluster across high latency networks, and suggests Federation instead. Did y
by m1keil 5y ago
Somewhat related question: Consul's deployment guide recommends against deploying a cluster across high latency networks, and suggests Federation instead. Did you had any issues caused by the high latency between servers and agents? Are all your Consul servers located in the same region?
- tptacek 5y agoThey are, and we do have those issues; we address them somewhat towards the end of the post, with São Paolo and Tokyo --- but also with São Paolo and São Paolo: a VM that pops up in GRU has to post to Consul in IAD before other GRU nodes "see" it in Consul. We've addressed that problem by "hinting", sending service discovery updates over NATS that usually beat out Consul, and then reconciling the two event sources. As I understand it, the reasonable-person thing to do here is to have federated Consul clusters in different regions. We run up against the problem that we're not hosting one application globally, but rather a zillion applications; every node needs to know about every instance of every application. So things like losing the global KV store are problematic for us.
- dilyevsky 5y agoSomething with “sharded consensus” like tikv or cockroach seems like a better fit based on this description
- mrkurt 5y agoWe actually don't have a consensus problem. Each "node" is its own source of truth and every other node can take what it says on faith. We _also_ need all the data in all the places.
- dilyevsky 5y agoThe “consensus” in raft is simply a way to guarantee durability of writes (among other things) in distributed setup. In your case the producer node gets a guarantee that it did in fact commit its route to the db. Ofc you can achieve similar outcome with something like gossip + repeated announcements. It would probably depend on specifics which approach is more efficient
- throwoutway 5y agoIAD? Are you hosting on another cloud provider or your own collocation?
- mrkurt 5y agoWe run our own servers. IAD is Ashburn, Virginia's airport code and there are a bunch of providers in Ashburn, Virginia.
- ignoramous 5y ago> We run up against the problem that we're not hosting one application globally, but rather a zillion applications; every node needs to know about every instance of every application. Have you considered nerd-sniping colmmacc? ;)
- EdwardDiego 5y agoTBH that's sound advice for pretty much every distributed system with a consensus protocol: high latency = slow consensus = slow change propagation.