3 ms·

This is incredible! I am curious about this paragraph though: > You’re going to see view changes when the primary crashes or is partitioned, and VSR’s telltale

This is incredible! I am curious about this paragraph though: > You’re going to see view changes when the primary crashes or is partitioned, and VSR’s telltale round robin rotation of the new primary among replicas, until a new primary is established. This is in contrast to Raft, which elects a primary at random, but then suffers from the risk of (or increased latency to mitigate) dueling leaders. It seems like regardless of primary/leader selection mechanism used (deterministic/roundrobin vs voting), you still need a quorum of nodes to agree (and for the minority side to know they can't proceed). Surely in a round robin selection mechanism some sort of vote or liveness check must be performed before it is safe for the #2 node to promote itself to #1/primary? Otherwise if the link between #2 and #1/primary is partitioned, #2 could unilaterally assume it was the primary/leader, even if the rest of the nodes could still communicate with #1 (the original primary). I don't understand how round robin solves the agreement aspect that leader election does. The simulation seems to only partition nodes and not links so I'm not sure it exercises asymmetric connectivity between members. Although it does mention flaky links, so perhaps they do cover this case. Edit: I have been informed there is still a vote. :)

jorangreef 3y ago
Thanks, great to hear you enjoyed it! The round robin "view change" in VSR is still consensus, and uses quorums to do fault isolation of the old primary, and to preserve the intersection property, to ensure that the committed log survives into the new view. What's cool about VSR's consensus though, is that the dice is also preloaded, ahead of time, so that there's more information baked into the protocol than with Raft or MultiPaxos, which means that you can neatly sidestep their dueling leader problem, or the latency padding that is often added to mitigate it. You can read more about VSR's intuitive view change consensus here: http://pmg.csail.mit.edu/papers/vr-revisited.pdf http://pmg.csail.mit.edu/papers/vr-revisited.pdf The round robin view chance can also make VSR (slightly) more resilient to weird network faults, like you describe. For example, variants of VSR using stable storage as hinted at by the '12 paper, are in fact able to survive most of the OmniPaxos liveness scenarios (we submitted some errata to the paper's authors for this). We haven't yet explored visualizing asymmetric connectivity in SimTigerBeetle (we tried to start with the big things!), however we recently wrote about how we test for this in our VOPR simulator here: https://tigerbeetle.com/blog/2023-07-06-simulation-testing-for-liveness/ https://tigerbeetle.com/blog/2023-07-06-simulation-testing-f...
- schmichael 3y ago
  Thanks for the details. A coworker pointed me at Heidi Howard (of Flexible Paxos, etc) and Diego Ongaro (Raft) discussing this very question! https://groups.google.com/g/raft-dev/c/cBNLTZT2q8o?pli=1 https://groups.google.com/g/raft-dev/c/cBNLTZT2q8o?pli=1
  - jorangreef 3y ago
    Yes, I think Heidi Howard gets it spot on there, in that the main difference is the view change. (TB implements Flexible Paxos and we're fans of Heidi Howard's work!)

jorangreef 3y ago

Thanks, great to hear you enjoyed it! The round robin "view change" in VSR is still consensus, and uses quorums to do fault isolation of the old primary, and to preserve the intersection property, to ensure that the committed log survives into the new view. What's cool about VSR's consensus though, is that the dice is also preloaded, ahead of time, so that there's more information baked into the protocol than with Raft or MultiPaxos, which means that you can neatly sidestep their dueling leader problem, or the latency padding that is often added to mitigate it. You can read more about VSR's intuitive view change consensus here: http://pmg.csail.mit.edu/papers/vr-revisited.pdf http://pmg.csail.mit.edu/papers/vr-revisited.pdf The round robin view chance can also make VSR (slightly) more resilient to weird network faults, like you describe. For example, variants of VSR using stable storage as hinted at by the '12 paper, are in fact able to survive most of the OmniPaxos liveness scenarios (we submitted some errata to the paper's authors for this). We haven't yet explored visualizing asymmetric connectivity in SimTigerBeetle (we tried to start with the big things!), however we recently wrote about how we test for this in our VOPR simulator here: https://tigerbeetle.com/blog/2023-07-06-simulation-testing-for-liveness/ https://tigerbeetle.com/blog/2023-07-06-simulation-testing-f...

schmichael 3y ago
Thanks for the details. A coworker pointed me at Heidi Howard (of Flexible Paxos, etc) and Diego Ongaro (Raft) discussing this very question! https://groups.google.com/g/raft-dev/c/cBNLTZT2q8o?pli=1 https://groups.google.com/g/raft-dev/c/cBNLTZT2q8o?pli=1
- jorangreef 3y ago
  Yes, I think Heidi Howard gets it spot on there, in that the main difference is the view change. (TB implements Flexible Paxos and we're fans of Heidi Howard's work!)