4 ms·

This is the first I have heard of Raft, but enjoyed the animations and ideas. I work on multi-node radio communications for ag automation. I had two questions a

This is the first I have heard of Raft, but enjoyed the animations and ideas. I work on multi-node radio communications for ag automation. I had two questions after watching this: - Is Raft alone in this space, or are there other popular algorithms/libraries that fill the same space? - What happens when the node count gets larger than a handful? What happens when you hit hundreds or even thousands of nodes, that are trying to achieve consensus? In particular, the part where all of the nodes respond (semi) simultaneously to a broadcast node. In a radio spectrum world, that would be a disaster. N:1 communication slots are choke points for timely communication.

throwawaymaths 4y ago
Paxos and viewstamped replication are basically the two most well known other well-known asynchronous consensus mechanisms that have been mathematically verified. If you just need eventual consistency, CRDTs are also possible. Going in the other direction, if you don't mind the latency full consensus with global locking, you could just do that.
aordano 4y ago
- There also is Paxos[0] as the most significant option. - You should not have too many nodes to make a decision, this is usually reserved for leaders; if you have a large distributed system you may clusterize them or forward decisions to leaders, whom decide for consensus. If you clusterize, the leaders for each node can also be selected by consensus. If you can't do any of those then having a consensus protocol might not even be a good idea; you'd end up with a sort of merkle tree (or some sort of blockchain) to make sure all the data is registered, or maybe audit transactions. In any case this[1] might be interesting. [0] https://en.wikipedia.org/wiki/Paxos_(computer_science) https://en.wikipedia.org/wiki/Paxos_(computer_science) [1] https://doi.org/10.1016/j.neucom.2016.10.011 https://doi.org/10.1016/j.neucom.2016.10.011
School-Cotton 4y ago
You don’t normally have hundreds or thousands of nodes trying to achieve consensus. You have 3-5 nodes trying to achieve consensus and then serving requests to the other 100s-1000s nodes.
woojoo666 4y ago
> - Is Raft alone in this space, or are there other popular algorithms/libraries that fill the same space? to give an obvious answer, blockchains are one of the methods used for trustless consensus (imagine how things could go wrong in RAFT's case if the leader was malicious)
- kjeetgill 4y ago
  Warning: you're going to get buried here. The 'right answer' is going to be Paxos and it's many flavors; a few references to Lamport, Google's Chubby, "Paxos Made Real". If the right people are on hn today you're going to get a few references ZAB (Zookeeper Atomic Broadcast) and View-Stampted Replication. Oh, and go watch all of Heidi Howard's talks. The problem spaces of byzantine and non-byzantine consensus are far enough apart ... well kinda like Atomicity in postgres and atomics is C++. You decide if that's a lot.
  - School-Cotton 4y ago
    I’m the biggest cryptocurrency hater there is but even I admit that, despite being mostly useless in the real world for any purpose other than committing crimes, Bitcoin solved a class of consensus problem that hadn’t been solved before and is therefore a major academic achievement. My GP probably shouldn’t have been downvoted.
    - kjeetgill 4y ago
      I agree completely! I always joke that the block and chain part of "blockchain" are boring: that's basically git! The cool part is byzantine consensus! But like I said, I also think that as a class of problems — byzantine consensus is just super different from the normal consensus problems paxos and raft are addressing. Way more than they might seem at first. Enough to be an off-topic to the point of just being naive or misleading.

throwawaymaths 4y ago

Paxos and viewstamped replication are basically the two most well known other well-known asynchronous consensus mechanisms that have been mathematically verified. If you just need eventual consistency, CRDTs are also possible. Going in the other direction, if you don't mind the latency full consensus with global locking, you could just do that.

aordano 4y ago

- There also is Paxos[0] as the most significant option. - You should not have too many nodes to make a decision, this is usually reserved for leaders; if you have a large distributed system you may clusterize them or forward decisions to leaders, whom decide for consensus. If you clusterize, the leaders for each node can also be selected by consensus. If you can't do any of those then having a consensus protocol might not even be a good idea; you'd end up with a sort of merkle tree (or some sort of blockchain) to make sure all the data is registered, or maybe audit transactions. In any case this[1] might be interesting. [0] https://en.wikipedia.org/wiki/Paxos_(computer_science) https://en.wikipedia.org/wiki/Paxos_(computer_science) [1] https://doi.org/10.1016/j.neucom.2016.10.011 https://doi.org/10.1016/j.neucom.2016.10.011

School-Cotton 4y ago

You don’t normally have hundreds or thousands of nodes trying to achieve consensus. You have 3-5 nodes trying to achieve consensus and then serving requests to the other 100s-1000s nodes.

woojoo666 4y ago

> - Is Raft alone in this space, or are there other popular algorithms/libraries that fill the same space? to give an obvious answer, blockchains are one of the methods used for trustless consensus (imagine how things could go wrong in RAFT's case if the leader was malicious)

kjeetgill 4y ago
Warning: you're going to get buried here. The 'right answer' is going to be Paxos and it's many flavors; a few references to Lamport, Google's Chubby, "Paxos Made Real". If the right people are on hn today you're going to get a few references ZAB (Zookeeper Atomic Broadcast) and View-Stampted Replication. Oh, and go watch all of Heidi Howard's talks. The problem spaces of byzantine and non-byzantine consensus are far enough apart ... well kinda like Atomicity in postgres and atomics is C++. You decide if that's a lot.
- School-Cotton 4y ago
  I’m the biggest cryptocurrency hater there is but even I admit that, despite being mostly useless in the real world for any purpose other than committing crimes, Bitcoin solved a class of consensus problem that hadn’t been solved before and is therefore a major academic achievement. My GP probably shouldn’t have been downvoted.
  - kjeetgill 4y ago
    I agree completely! I always joke that the block and chain part of "blockchain" are boring: that's basically git! The cool part is byzantine consensus! But like I said, I also think that as a class of problems — byzantine consensus is just super different from the normal consensus problems paxos and raft are addressing. Way more than they might seem at first. Enough to be an off-topic to the point of just being naive or misleading.