4 ms·

> Raft doesn't deal with the faults I'm not sure what this means. The Raft paper explicitly states that Raft is a fault-tolerant system -- and by definition th

> Raft doesn't deal with the faults I'm not sure what this means. The Raft paper explicitly states that Raft is a fault-tolerant system -- and by definition this means it deals with faults. To quote the paper: "Replicated state machines are used to solve a variety of fault tolerance problems in distributed systems." Raft is a type of replicated state machine. To say that "Raft doesn't deal with faults" is not correct. Perhaps you mean that there is other work to be done to take the fault tolerance offered by Raft and build a functioning application, and a system that will stay up in the real world -- and deal with an even wider range of faults. That I definitely agree with.

heavenlyblue 9y ago
Well, if you’re being smart - do please tell us how Raft solves the problem of online faulty node replacement: how how does it re-sync the current, up-to-date state to the new node, how does it make sure that when one of the nodes fails - that all of the clients do not overwhelm the rest of the system. As far as Google went with their paper; they were incredibly correct by saying that actually implementing a distributed system is hard because of the failure modes; rather than the success modes of the running system.
- otoolep 9y ago
  >Well, if you’re being smart - do please tell us how Raft solves the problem of online faulty node replacement: I never said that it did. That is what I mean by my statement that more work is required to build a real system in the real world. Much of the code of rqlite is about cluster management, built on a Raft substrate. >how how does it re-sync the current, up-to-date state to the new node I suggest you read the Raft paper, and study the Hashicorp Go implementation (https://github.com/hashicorp/raft https://github.com/hashicorp/raft) to see how this is done. It's all there. >As far as Google went with their paper; they were incredibly correct by saying that actually implementing a distributed system is hard because of the failure modes; rather than the success modes of the running system. I couldn't agree more.
- free-ekanayaka 9y ago
  If you read the raft paper, or just watch presentations of it, you'll see that all the issues you mentioned are covered: - faulty node replacement: you can take any node off (or any node can crash) at any time. As long as there are enough nodes left to reach a quorum, your system will be available. If there are not enough nodes left, your system will be unavailable (but keep consistency) - re-syncs, snapshots and the rest are all covered in the raft paper - clients wanting to perform writes always talk to the node that is currently the leader, that node fails, clients will look for next leader (which will be eventually elected as long as there's a quorum of surviving nodes) - overwhelming a system is a different concern, raft writes are serialized so the goal is usually not throughput (for that you might look at AP/AC storage solutions in the CAP spectrum, raft is CP). Designs that need high write throughput might still use raft as internal building block for coordination.
- dis-sys 9y ago
  > how does it re-sync the current, up-to-date state to the new node it is a core feature of the protocol. > how does it make sure that when one of the nodes fails - that all of the clients do not overwhelm the rest of the system. for write access (proposals), in a typical implementation (defined below [1]), one failed nodes actually slightly speed up the system with the cost of reduced reliability. Consider a typical setup with 3 nodes, the leader normally replicate state to 2 followers, the replication cost get cut in half once your cluster loses one member. given that you can implement linearizable read by going through the write procedure, one can argue the above "speed up" can obviously be achieved on reads as well. [1] independent replication to followers, for N followers, the same entry will be serialised and sent N times by the leader. again - just talking about typical implementation and the described "speed up" comes with degraded reliability.
otoolep 9y ago
Rereading this, I see what you mean in the sense that if dealing == "fixing the fault", but tolerating == "keep going in the face of faults". So yes, you are right -- Raft doesn't fix the faults. One must still fix the network, replace the node etc. When I wrote "deal" I meant "tolerate the fault", but I think "deal" meant "fix the fault" to you.

heavenlyblue 9y ago

Well, if you’re being smart - do please tell us how Raft solves the problem of online faulty node replacement: how how does it re-sync the current, up-to-date state to the new node, how does it make sure that when one of the nodes fails - that all of the clients do not overwhelm the rest of the system. As far as Google went with their paper; they were incredibly correct by saying that actually implementing a distributed system is hard because of the failure modes; rather than the success modes of the running system.

otoolep 9y ago
>Well, if you’re being smart - do please tell us how Raft solves the problem of online faulty node replacement: I never said that it did. That is what I mean by my statement that more work is required to build a real system in the real world. Much of the code of rqlite is about cluster management, built on a Raft substrate. >how how does it re-sync the current, up-to-date state to the new node I suggest you read the Raft paper, and study the Hashicorp Go implementation (https://github.com/hashicorp/raft https://github.com/hashicorp/raft) to see how this is done. It's all there. >As far as Google went with their paper; they were incredibly correct by saying that actually implementing a distributed system is hard because of the failure modes; rather than the success modes of the running system. I couldn't agree more.
free-ekanayaka 9y ago
If you read the raft paper, or just watch presentations of it, you'll see that all the issues you mentioned are covered: - faulty node replacement: you can take any node off (or any node can crash) at any time. As long as there are enough nodes left to reach a quorum, your system will be available. If there are not enough nodes left, your system will be unavailable (but keep consistency) - re-syncs, snapshots and the rest are all covered in the raft paper - clients wanting to perform writes always talk to the node that is currently the leader, that node fails, clients will look for next leader (which will be eventually elected as long as there's a quorum of surviving nodes) - overwhelming a system is a different concern, raft writes are serialized so the goal is usually not throughput (for that you might look at AP/AC storage solutions in the CAP spectrum, raft is CP). Designs that need high write throughput might still use raft as internal building block for coordination.
dis-sys 9y ago
> how does it re-sync the current, up-to-date state to the new node it is a core feature of the protocol. > how does it make sure that when one of the nodes fails - that all of the clients do not overwhelm the rest of the system. for write access (proposals), in a typical implementation (defined below [1]), one failed nodes actually slightly speed up the system with the cost of reduced reliability. Consider a typical setup with 3 nodes, the leader normally replicate state to 2 followers, the replication cost get cut in half once your cluster loses one member. given that you can implement linearizable read by going through the write procedure, one can argue the above "speed up" can obviously be achieved on reads as well. [1] independent replication to followers, for N followers, the same entry will be serialised and sent N times by the leader. again - just talking about typical implementation and the described "speed up" comes with degraded reliability.

otoolep 9y ago

Rereading this, I see what you mean in the sense that if dealing == "fixing the fault", but tolerating == "keep going in the face of faults". So yes, you are right -- Raft doesn't fix the faults. One must still fix the network, replace the node etc. When I wrote "deal" I meant "tolerate the fault", but I think "deal" meant "fix the fault" to you.