3 ms·

What Aphyr tested was not my "toy" example model, that was not even proposed as something to actually implement, but just to show that WAIT per se is not broken

by antirez 13y ago

What Aphyr tested was not my "toy" example model, that was not even proposed as something to actually implement, but just to show that WAIT per se is not broken or ok, it is just a low level building block. The consistency achieved depends on the whole system, especially the failover procedure safety guarantees. What I proposed is a toy system as described here: https://gist.github.com/antirez/7901666 https://gist.github.com/antirez/7901666 It is a toy as it has a super strong coordinator that can partition away instances, that is never partitioned, that can reconfigure clients magically and so forth. Under the above assumptions the theoretical system is trivially capable of reaching linearizability I believe. Aphyr tested a different model, with an implementation that is not capable to even guarantee some weak assumption about the model (for example, the slave reset the replication offset to 0 when restarted), so I'm not sure what the result means. I could test the actual model I proposed even with Redis, but manually following the steps that I outlined in the Redis mailing list thread. The point was, if you can guarantee certain properties in the system, there is always the "transfer" of data and higher offsets to a majority of replicas, and the system becomes strongly consistent. The properties are hard to achieve in practice once you try to move the features from the mythical super strong coordinator into the actual system, and this is why, for example, Raft uses epochs and other mechanisms to guarantee both safety and liveness. Unfortunately the focus is in showing other people are wrong without even caring where the discussion is headed. --- EDIT --- Btw now that I read the full post carefully, Aphyr also cherry-picked parts of the thread to construct a story that does not exist, like if I was going to implement strong consistency into Redis based on the proposed toy system that was only useful to show that WAIT per se was not a system, but just a building block. Note that yesterday I wrote the opposite in my blog, that there is no interest in strong consistency in Redis Cluster. Very unfair IMHO... I read only the analysis part at first, and was thinking this was just a "let's check anyway this model with the current implementation".

mmcnickle 13y ago
So to clarify, you see WAIT as a replication primative. It can be used as part of a larger scheme with a strong coordinator to provide strong consistency.
- antirez 13y ago
  Yes, with a strong coordinator or with a distributed system that can provide similar guarantees. But currently there is no plan to add this in Redis Cluster. Three main reasons: * Our main business is low latency, so very few will use synchronous replication. * Redis cluster is composed of multiple master-slave systems that hold a subset of the key space each. This means that if you need a majority of replicas to promote a new master in order to achieve consistency, after a partition you end with different hash slots having the majority in different sides of the partitions perhaps. * With the current weaker consistency guarantees Redis Cluster can elect a replica that is isolated by the others but is in the right side of the partition, where the majority of other masters are. Similarly you can have just two total nodes for every hash slot, a setup that I believe will be very used, and still get some availability if the master fails. So the consistency premises and the tradeoff of Redis Cluster are not exactly compatible with strong consistency. Is WAIT still a useful tool? I believe yes, if documented as such: WAIT in the context of Redis Cluster / Sentinel is not able to provide strong consistency, but lowers the percentage of probabilities that a failure mode that results in data loss happens. A trivial example, what happens if a client is partitioned away with just a master? Without WAIT there is a window of NODE_TIMEOUT to lose data, with WAIT there is not this problem in this specific failure scenario. There are still a number of failure modes for WAIT, but are a smaller number compared to the full set of failure modes that there are without WAIT, so practically it does not feature strong consistency but provides the user with a smaller probability of data loss. Another example: manual failover for very important data of some kind. You may have 5 nodes and use WAIT 4 to always write everywhere and improve durability. And so forth.

by antirez 13y ago

mmcnickle 13y ago

So to clarify, you see WAIT as a replication primative. It can be used as part of a larger scheme with a strong coordinator to provide strong consistency.

antirez 13y ago
Yes, with a strong coordinator or with a distributed system that can provide similar guarantees. But currently there is no plan to add this in Redis Cluster. Three main reasons: * Our main business is low latency, so very few will use synchronous replication. * Redis cluster is composed of multiple master-slave systems that hold a subset of the key space each. This means that if you need a majority of replicas to promote a new master in order to achieve consistency, after a partition you end with different hash slots having the majority in different sides of the partitions perhaps. * With the current weaker consistency guarantees Redis Cluster can elect a replica that is isolated by the others but is in the right side of the partition, where the majority of other masters are. Similarly you can have just two total nodes for every hash slot, a setup that I believe will be very used, and still get some availability if the master fails. So the consistency premises and the tradeoff of Redis Cluster are not exactly compatible with strong consistency. Is WAIT still a useful tool? I believe yes, if documented as such: WAIT in the context of Redis Cluster / Sentinel is not able to provide strong consistency, but lowers the percentage of probabilities that a failure mode that results in data loss happens. A trivial example, what happens if a client is partitioned away with just a master? Without WAIT there is a window of NODE_TIMEOUT to lose data, with WAIT there is not this problem in this specific failure scenario. There are still a number of failure modes for WAIT, but are a smaller number compared to the full set of failure modes that there are without WAIT, so practically it does not feature strong consistency but provides the user with a smaller probability of data loss. Another example: manual failover for very important data of some kind. You may have 5 nodes and use WAIT 4 to always write everywhere and improve durability. And so forth.