4 ms·
Raft involves waiting for fsync on a majority of nodes, so that's not too surprising. 'Typical' is a matter of what guarantees you want to give.
by refenestrator 5y ago
Raft involves waiting for fsync on a majority of nodes, so that's not too surprising.
'Typical' is a matter of what guarantees you want to give.
- toolz 5y agoTypically people use raft for leader election which in turn can coordinate writes. I don't think the writes are being fsync'd in the raft logs here. At least I wouldn't expect that behavior.
- alexchamberlain 5y agoEach write should be fsync'd to the WAL, right?
- stingraycharles 5y agoYes but those can happen at the convenience of the particular node, not necessarily as a globally chrckpointed fsync()
- refenestrator 5y agoThen you're sacrificing consistency guarantees. If less than a majority have committed a write, it could be lost while the cluster still has a quorum up. Waiting to report success until a majority have committed allows you to make guarantees with a straight face.. "it will probably be committed in the near future" is not the same thing.
- rubiquity 5y agoNot necessarily. As long as your quorums always overlap you can have weaker commit requirements. See the FPaxos paper and some of Heidi Howard’s blogs for more on this. https://fpaxos.github.io/ https://fpaxos.github.io/
- jimsimmons 5y agoFPaxos requires commit quorum and subsequent promise request quorum to intersect. It doesn't bring any significant thruput benefits
- rubiquity 5y agoI didn't say FPaxos brought throughput benefits. I pointed out that you can still have a correct implementation without requiring majority quorums. But since you brought it up, FPaxos can have both lower latency and higher throughput than MultiPaxos/Raft[0] ;) 0 - https://dl.acm.org/doi/pdf/10.1145/3299869.3319893 https://dl.acm.org/doi/pdf/10.1145/3299869.3319893
- jimsimmons 5y agoGreat link. Need to look into what WPaxos is.
- jimsimmons 5y agoYou don't understand Raft. Quorum has to fsync for commit
- rubiquity 5y agoIt reads like you and GP are talking about two different things. You’re talking about Raft specifics and the GP seems to be talking about a Vertical Paxos like setup where Raft is used for configuration and the data path uses another replication algorithm such as Primary-Backup or Chain Replication.
- jimsimmons 5y agoHmmm that sorta makes sense I guess. Sorta because raft is a replication algorithm. If you don't use raft in datapath you don't get any of its guarantees
- rubiquity 5y agoMost data paths don't need consensus for replication. You can implement ACID Transactions on top of many other replication algorithms for example. Ultimately the choice comes down to read/write ratios. Chain Replication has much better read throughout than Paxos and Raft.
- jimsimmons 5y agoI see. What's the catch though. Sounds like free lunch. Is there some gotcha with partition tolerance?
- rubiquity 5y agoIt’s the case of 2F+1 versus F+1. Paxos/Raft offer fault tolerance where as other replication algorithms don’t. If you have a node failure in Primary-Backup or Chain for example you need to reconfigure before committing more writes. However in practice and certain environments reconfiguration can be faster than or the same as a leader failure in Paxos that requires running Phase 1 again.
- skyde 5y agoFor the raft algorithm to be correct Fsync is required on a majority of node otherwise you are technically not implementing Raft. The reason is that in Raft if a node acknowledges to the leader that it wrote something to the log it must not later accept a different write in the same log position. This mean if for some reason server rebooted with dirty buffered writes that could not be flushed in time. it’s supposed to forgot everything it know and rejoin the cluster using a brand new node id.