29 ms·

The moment you see another leader with higher epoch you cecede and become a follower. This should be part of message recieve logic and has highest priority. Any

The moment you see another leader with higher epoch you cecede and become a follower. This should be part of message recieve logic and has highest priority. Any actions you might take are moot anyway because higher epoch leader always the source of truth

valzam 5y ago
That is what I was thinking. But does that mean you just pepper your command processing logic with guards that you are still actually the leader? I think one mistake I made is trying to run the different parts of the protocol in parallel, i.e. processing a message, accepting messages, sending heartbeats etc. It probably makes more sense for the raft part to basically be single threaded.
- jimsimmons 5y ago
  > That is what I was thinking. But does that mean you just pepper your command processing logic with guards that you are still actually the leader? Yep. It's annoying to code it this way but you have to. One way to make this relatively clean is via throwing and catching exceptions if your language has good support for it. Also agree re single-threadedness. In some ways raft assumes a single state machine loop running on a node at any point. Concurrency introduces additional non determinism to the state machine and isn't really accounted for in the formal spec. I've heard great things about etcd source and how clean it is. I've never studied it but it might give you some pointers on best practices.
- _vvhw 5y ago
  If you can avoid multithreading in the design of your consensus protocol, this also means that your implementation becomes more deterministic. If you can have the design fully deterministic by stubbing out the message passing, storage and time source (by representing time as ticks), then you can do deterministic simulation fuzzing like we do in TigerBeetle, which is a high velocity testing technique for finding/reproducing/fixing interesting (and rare) distributed systems bugs: https://github.com/coilhq/tigerbeetle#simulation-tests https://github.com/coilhq/tigerbeetle#simulation-tests
- tubby12345 5y ago
  My implementation is multithreaded and parallel (ie I have multiple cores). The way I handle this is to have the leader check if it's still the leader before it does anything. That flag (`iamtheleader`) is set and read behind a mutex. I guess there's a race condition (old leader initiates read right before rpc sets that leader has ceded) but I never hit it. I guess the right way to do it is that the rpc that handles ceding should kill other rpc threads but that seems fraught. I wonder if there's some kind of interrupt request handler type thing in threading models...

valzam 5y ago

That is what I was thinking. But does that mean you just pepper your command processing logic with guards that you are still actually the leader? I think one mistake I made is trying to run the different parts of the protocol in parallel, i.e. processing a message, accepting messages, sending heartbeats etc. It probably makes more sense for the raft part to basically be single threaded.

jimsimmons 5y ago
> That is what I was thinking. But does that mean you just pepper your command processing logic with guards that you are still actually the leader? Yep. It's annoying to code it this way but you have to. One way to make this relatively clean is via throwing and catching exceptions if your language has good support for it. Also agree re single-threadedness. In some ways raft assumes a single state machine loop running on a node at any point. Concurrency introduces additional non determinism to the state machine and isn't really accounted for in the formal spec. I've heard great things about etcd source and how clean it is. I've never studied it but it might give you some pointers on best practices.
_vvhw 5y ago
If you can avoid multithreading in the design of your consensus protocol, this also means that your implementation becomes more deterministic. If you can have the design fully deterministic by stubbing out the message passing, storage and time source (by representing time as ticks), then you can do deterministic simulation fuzzing like we do in TigerBeetle, which is a high velocity testing technique for finding/reproducing/fixing interesting (and rare) distributed systems bugs: https://github.com/coilhq/tigerbeetle#simulation-tests https://github.com/coilhq/tigerbeetle#simulation-tests
tubby12345 5y ago
My implementation is multithreaded and parallel (ie I have multiple cores). The way I handle this is to have the leader check if it's still the leader before it does anything. That flag (`iamtheleader`) is set and read behind a mutex. I guess there's a race condition (old leader initiates read right before rpc sets that leader has ceded) but I never hit it. I guess the right way to do it is that the rpc that handles ceding should kill other rpc threads but that seems fraught. I wonder if there's some kind of interrupt request handler type thing in threading models...