3 ms·

ah ok, so you mean the "node membership is homogenous". i thought you were talking about the transport layer.

• hinkley 7y ago
  I think we might still be missing each other. Everybody has to be able to see everybody. Everyone sends their updates to the leader. The leader sends those to everyone else. When the leader dies, consensus is used to elect a new leader, based on 1) who asks first, and 2) whether everyone agrees they have the latest stuff. That is the consensus moment. It is possible for one server in a remote data center to have enough inbound bandwidth to be 100% up to date when the election happens, but be throttled for outbound traffic. It may get itself elected and then be voted out the very next time that a spike of traffic occurs. If you have a couple of these, they could spend all day swapping around leadership. And to be crystal clear: I'm not talking hypotheticals here. I'm talking about corner cases that people have had to confront in real systems. I brought up the case of Consul earlier. I could have a bunch of servers locally, harassing a local Consul server to avoid making higher-latency requests over a potentially constrained interface from, for instance, a branch office. You wouldn't want any of the branch offices to ever be leader unless the home office is completely offline. One of the 3 servers located there should always be online, even in a double fault situation (eg, someone is upgrading one server and restarts the wrong one).
  • dnautics 7y ago
    Cool, that's a great example, and I see exactly why it's a problem (I've implemented raft). Thank you for clarifying.