3 ms·

Can Raft scale to 100s of nodes? It doesn't seem likely, or at least, I'm not seeing people trying to. I suppose one can have a subset of a cluster participati

Can Raft scale to 100s of nodes? It doesn't seem likely, or at least, I'm not seeing people trying to. I suppose one can have a subset of a cluster participating in Raft, but then the "raft group membership" itself is a consensus problem isn't it? It's all turtles...

hawk_ 5y ago
since each node is a replica, 100s of replica nodes would be wasteful. for scaling say something like 100s of "shards" where each shard is a raft cluster of 3-5 nodes is more appropriate.
- butterisgood 5y ago
  It depends really... If you need an "offline readable copy" of something, then you need to be able to enter and exit the consensus group somehow. Perhaps that's replaceable though by a client/server relationship with the Raft cluster and keep things separate? (Probably, though one of the systems I've worked with didn't have that as an option - though it likely could have been done as such). This was only really feasible because the value under consensus was very small (far less than 1MB). In a system where the value(s) are much larger, I think what you've proposed as a system of Raft clusters hosting shards makes a lot of sense too. Thanks for the considerate answer!
teraflop 5y ago
> I suppose one can have a subset of a cluster participating in Raft, but then the "raft group membership" itself is a consensus problem isn't it? Yes, this is discussed briefly in the article and more thoroughly in the original Raft paper. It's possible to do a "membership change" operation, whereby the replicas agree to change the set of nodes in the cluster. It's slightly trickier than achieving consensus on an ordinary state machine operation, because you have to ensure that quorums from both the old and new node lists agree to the change. With this feature, you can make a "self-healing" cluster. Say you have a pool of 100 physical machines, with 5 of them running Raft nodes. If one node fails, and the failure appears to be transient, you can just wait for it to come back up. Otherwise, you can start a new node, wait for it to come up, and perform a membership change to swap it for the failed one. This provides availability as long as you don't have a burst of more than N/2 failures faster than the cluster can recover.
- butterisgood 5y ago
  Thanks! This is a great, thought-provoking answer. I've only worked with variants of paxos which were greatly simplified, and wondered how Raft actually simplified things here. I feel like Raft, despite being easier to explain quickly, really doesn't seem much simpler than a basic explanation of Paxos. But perhaps that's because of what I've already worked with. I've got an itch to explore Viewstamped Replication as well...

hawk_ 5y ago

since each node is a replica, 100s of replica nodes would be wasteful. for scaling say something like 100s of "shards" where each shard is a raft cluster of 3-5 nodes is more appropriate.

butterisgood 5y ago
It depends really... If you need an "offline readable copy" of something, then you need to be able to enter and exit the consensus group somehow. Perhaps that's replaceable though by a client/server relationship with the Raft cluster and keep things separate? (Probably, though one of the systems I've worked with didn't have that as an option - though it likely could have been done as such). This was only really feasible because the value under consensus was very small (far less than 1MB). In a system where the value(s) are much larger, I think what you've proposed as a system of Raft clusters hosting shards makes a lot of sense too. Thanks for the considerate answer!

teraflop 5y ago

> I suppose one can have a subset of a cluster participating in Raft, but then the "raft group membership" itself is a consensus problem isn't it? Yes, this is discussed briefly in the article and more thoroughly in the original Raft paper. It's possible to do a "membership change" operation, whereby the replicas agree to change the set of nodes in the cluster. It's slightly trickier than achieving consensus on an ordinary state machine operation, because you have to ensure that quorums from both the old and new node lists agree to the change. With this feature, you can make a "self-healing" cluster. Say you have a pool of 100 physical machines, with 5 of them running Raft nodes. If one node fails, and the failure appears to be transient, you can just wait for it to come back up. Otherwise, you can start a new node, wait for it to come up, and perform a membership change to swap it for the failed one. This provides availability as long as you don't have a burst of more than N/2 failures faster than the cluster can recover.

butterisgood 5y ago
Thanks! This is a great, thought-provoking answer. I've only worked with variants of paxos which were greatly simplified, and wondered how Raft actually simplified things here. I feel like Raft, despite being easier to explain quickly, really doesn't seem much simpler than a basic explanation of Paxos. But perhaps that's because of what I've already worked with. I've got an itch to explore Viewstamped Replication as well...