3 ms·

> Raft, and I believe Panos, contain some of the 8 Fallacies of Distributed Computing. The most unavoidable being assuming the network is heterogenous. Did you

> Raft, and I believe Panos, contain some of the 8 Fallacies of Distributed Computing. The most unavoidable being assuming the network is heterogenous. Did you mean homogeneous? Also, Raft doesn't assume the network is homogeneous. > I’ve heard of Raft variants with three (pure consumers, full voting members, and new or returning members who are still catching up) so that you can use machines unprepared for leadership and they will never try to elect themselves leader. What's the purpose of these non-leader-competent nodes? Is it merely to increase the quorum membership and make consistency more difficult? Why wouldn't you architect this as a client/server relationship?

hinkley 7y ago
> Did you mean homogeneous? Yes > Also, Raft doesn't assume the network is homogeneous. Raft assumes that if the leader dies, that quick and orderly election relies on all members doing their 'duty' and putting themselves forward as the new leader if some counter says they are next. That means it assumes that any server can be leader. What is implicit in this state of affairs is that all cluster members are equipped to perform the duties of the leader, which includes rebroadcasting all state changes - directly - to all other members. So every single machine has to have enough bandwidth to every other machine to comfortably ferry all of that data across. That is a huge assumption, both historically speaking, and with geographical distribution of servers.
- dnautics 7y ago
  ah ok, so you mean the "node membership is homogenous". i thought you were talking about the transport layer.
  - hinkley 7y ago
    I think we might still be missing each other. Everybody has to be able to see everybody. Everyone sends their updates to the leader. The leader sends those to everyone else. When the leader dies, consensus is used to elect a new leader, based on 1) who asks first, and 2) whether everyone agrees they have the latest stuff. That is the consensus moment. It is possible for one server in a remote data center to have enough inbound bandwidth to be 100% up to date when the election happens, but be throttled for outbound traffic. It may get itself elected and then be voted out the very next time that a spike of traffic occurs. If you have a couple of these, they could spend all day swapping around leadership. And to be crystal clear: I'm not talking hypotheticals here. I'm talking about corner cases that people have had to confront in real systems. I brought up the case of Consul earlier. I could have a bunch of servers locally, harassing a local Consul server to avoid making higher-latency requests over a potentially constrained interface from, for instance, a branch office. You wouldn't want any of the branch offices to ever be leader unless the home office is completely offline. One of the 3 servers located there should always be online, even in a double fault situation (eg, someone is upgrading one server and restarts the wrong one).
    - dnautics 7y ago
      Cool, that's a great example, and I see exactly why it's a problem (I've implemented raft). Thank you for clarifying.
grogers 7y ago
Regarding the other types of nodes, pure consumers are useful as "standby" nodes. They are ready to be swapped in when another host fails, but don't participate in voting until they are swapped in. Swapping them in still requires a quorum of the previous nodes, so they are only useful in small numbers to be swapped in when another node fails. Nodes that are catching up are just a special case of this pure consumer type that shouldn't be promoted to be a full voting member until they are fully caught up. I'll add another type of node that sometimes comes in handy in certain network topologies. I've typically heard this called a "witness" - it is a voting member but doesn't store any actual data (just metadata) so it can't be a leader. The typical use for this is if you have a data center cluster with only two DCs. In that scenario you can make the number of nodes in each DC unbalanced so if the minority DC goes down you still have a quorum. But you can't make it so that if either DC goes down you can still have a quorum. So you add a third DC over a WAN that acts as an arbitrator between the two. Because it's far away, you don't want it to store all the data - keeping it in sync would be expensive. You only need it to know enough metadata to vote correctly. This obviously changes the failure characteristics quite a bit vs having 3 DCs in the cluster like normal. Such as how many real nodes and witness nodes you can tolerate failing. Latency while one of your your main DCs are down is much higher than normal (because you now need acks from the witnesses) but you at least keep running.
- dnautics 7y ago
  Do these alternative topologies still have the proven guarantees that raft has?
- senderista 7y ago
  See Cheap Paxos.

hinkley 7y ago

> Did you mean homogeneous? Yes > Also, Raft doesn't assume the network is homogeneous. Raft assumes that if the leader dies, that quick and orderly election relies on all members doing their 'duty' and putting themselves forward as the new leader if some counter says they are next. That means it assumes that any server can be leader. What is implicit in this state of affairs is that all cluster members are equipped to perform the duties of the leader, which includes rebroadcasting all state changes - directly - to all other members. So every single machine has to have enough bandwidth to every other machine to comfortably ferry all of that data across. That is a huge assumption, both historically speaking, and with geographical distribution of servers.

dnautics 7y ago
ah ok, so you mean the "node membership is homogenous". i thought you were talking about the transport layer.
- hinkley 7y ago
  I think we might still be missing each other. Everybody has to be able to see everybody. Everyone sends their updates to the leader. The leader sends those to everyone else. When the leader dies, consensus is used to elect a new leader, based on 1) who asks first, and 2) whether everyone agrees they have the latest stuff. That is the consensus moment. It is possible for one server in a remote data center to have enough inbound bandwidth to be 100% up to date when the election happens, but be throttled for outbound traffic. It may get itself elected and then be voted out the very next time that a spike of traffic occurs. If you have a couple of these, they could spend all day swapping around leadership. And to be crystal clear: I'm not talking hypotheticals here. I'm talking about corner cases that people have had to confront in real systems. I brought up the case of Consul earlier. I could have a bunch of servers locally, harassing a local Consul server to avoid making higher-latency requests over a potentially constrained interface from, for instance, a branch office. You wouldn't want any of the branch offices to ever be leader unless the home office is completely offline. One of the 3 servers located there should always be online, even in a double fault situation (eg, someone is upgrading one server and restarts the wrong one).
  - dnautics 7y ago
    Cool, that's a great example, and I see exactly why it's a problem (I've implemented raft). Thank you for clarifying.

grogers 7y ago

Regarding the other types of nodes, pure consumers are useful as "standby" nodes. They are ready to be swapped in when another host fails, but don't participate in voting until they are swapped in. Swapping them in still requires a quorum of the previous nodes, so they are only useful in small numbers to be swapped in when another node fails. Nodes that are catching up are just a special case of this pure consumer type that shouldn't be promoted to be a full voting member until they are fully caught up. I'll add another type of node that sometimes comes in handy in certain network topologies. I've typically heard this called a "witness" - it is a voting member but doesn't store any actual data (just metadata) so it can't be a leader. The typical use for this is if you have a data center cluster with only two DCs. In that scenario you can make the number of nodes in each DC unbalanced so if the minority DC goes down you still have a quorum. But you can't make it so that if either DC goes down you can still have a quorum. So you add a third DC over a WAN that acts as an arbitrator between the two. Because it's far away, you don't want it to store all the data - keeping it in sync would be expensive. You only need it to know enough metadata to vote correctly. This obviously changes the failure characteristics quite a bit vs having 3 DCs in the cluster like normal. Such as how many real nodes and witness nodes you can tolerate failing. Latency while one of your your main DCs are down is much higher than normal (because you now need acks from the witnesses) but you at least keep running.

dnautics 7y ago
Do these alternative topologies still have the proven guarantees that raft has?
senderista 7y ago
See Cheap Paxos.