3 ms·
When a VM comes up, we write a service to consul and simultaneously broadcast an identical NATS message. Nodes that are in a happy place get the NATS message, u
by mrkurt 5y ago
When a VM comes up, we write a service to consul and simultaneously broadcast an identical NATS message. Nodes that are in a happy place get the NATS message, update their state, and then reconcile against Consul when Consul catches up.
NATS has the benefit of being p2p, so when a VM comes up in Sydney, the other Sydney nodes see:
- The nats message in about 5ms
- Time passes (call it 1-10s when things are working well)
- Consul data includes new entry
We also use "hints" to broadcast things that we can afford to never see. Load on VMs is an example. When a VM hits its max concurrency, we broadcast a NATS message so other nodes can take that into account for load balancing.
- kasey_junk 5y agoDo you view Consul as authoritative once it arrives if it clashes with the hint?
- mrkurt 5y agoMostly yes. There are some races, particularly when services are deleted. A delete hint is actually authoritative.
- kasey_junk 5y agoAwesome. Thanks!
- tptacek 5y agoIt's complicated; there's some reconciliation logic that considers the type of update and the timing of the hint vs. the Consul status. When a VM startup hint lands, beating Consul, Consul will continuously assert the non-existence of that VM until the whole Raft dance completes and the API polling picks it up --- so, effectively, the whole point of the hints is not to let Consul be authoritative for some data, for short periods of time. It's super gross! The basic use case here is: you start or stop a VM in Tokyo; our Consul Server cluster is halfway around the world, and every other host in Tokyo needs to wait for Consul to update to see that change, even though it happened right next door. With the hints, the change percolates instantly, and eventually Consul catches up and we reconcile.