3 ms·

Have you dealt with mitigating Asymmetric partitions in Raft? http://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-857.pdf http://www.cl.cam.ac.uk/techreports/UCAM-CL

Have you dealt with mitigating Asymmetric partitions in Raft? http://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-857.pdf http://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-857.pdf shows that a single defective node has the potential to entirely render a Raft cluster useless, specifically in the case of asymmetric partitions: > The mobile devices will often be disconnected from the majority of other devices, thus running election after election without being able to gain enough votes. When these devices rejoin the network, they will force a new election as their term will be much higher than the leader. This leads to frequently disconnected devices becoming the leader when reattached to the network, thus causing regular unavailability of the system when they leave again. > [...] > It is also interesting to consider the impact of a single node being behind an asymmetric partition [...] This node will repeatedly re-run elections as it will not be able to receive incoming messages, each RequestVotes will have a higher term than the last, forcing the leader to step down. Specifically, they required to add client request caches, exposing serial transaction IDs in every single record and log entry, specific timeouts and measures for livelock bugs this introduced, which include replacing an entry by a noop value to be replicated on some read-only operation done at a later time. On top of this, these modifications may end up reducing the serializability constructs to bring it closer to what a Master-Follower replication may give in a DB -- i.e. it doesn't guarantee one true serializable sequence of commits anymore, but only that the state replication will be consistent across all nodes

danielmewes 11y ago
We haven't added special logic for handling such cases. I think having servers with particularly unstable connectivity in a RethinkDB cluster is really uncommon though. If you want to run a RethinkDB server on your mobile device that's a different scenario of course...
- mononcqc 11y ago
  Well the problem is obvious in mobile devices, but any asymmetric netsplit will result in a single node taking down your cluster for as long as the netsplit lasts.
  - danielmewes 11y ago
    Ah, I have to check how we behave in that case. Thanks for the pointers.
    - danielmewes 11y ago
      If I understand that scenario correctly, it assumes that there is a node that can successfully send messages to the other nodes in the cluster, but cannot receive any responses. Since RethinkDB uses TCP connections, this shouldn't usually happen (since the TCP acknowledgements wouldn't get through either). The exception might be a layer 5 router / firewall somewhere in the network that allows the TCP connection to work, but only passes the data stream through in one direction. RethinkDB is partially protected against this case, because we use bidirectional heartbeats on top of the same TCP connection that is used for Raft traffic. The heartbeat usually ensures that the other host is still alive and reachable. In this case, the node that cannot receive any responses from the remaining cluster would get a heartbeat timeout after a couple of seconds, and disconnect from the remaining servers. This is turn should limit further damage and allow the Raft cluster to proceed. Please let me know if I'm missing something else. Edit: There might still be some problems with non-transitive connectivity, where one node can talk to only parts of the cluster. We have built in some protection against this, but don't always handle that case well.

danielmewes 11y ago

We haven't added special logic for handling such cases. I think having servers with particularly unstable connectivity in a RethinkDB cluster is really uncommon though. If you want to run a RethinkDB server on your mobile device that's a different scenario of course...

mononcqc 11y ago
Well the problem is obvious in mobile devices, but any asymmetric netsplit will result in a single node taking down your cluster for as long as the netsplit lasts.
- danielmewes 11y ago
  Ah, I have to check how we behave in that case. Thanks for the pointers.
  - danielmewes 11y ago
    If I understand that scenario correctly, it assumes that there is a node that can successfully send messages to the other nodes in the cluster, but cannot receive any responses. Since RethinkDB uses TCP connections, this shouldn't usually happen (since the TCP acknowledgements wouldn't get through either). The exception might be a layer 5 router / firewall somewhere in the network that allows the TCP connection to work, but only passes the data stream through in one direction. RethinkDB is partially protected against this case, because we use bidirectional heartbeats on top of the same TCP connection that is used for Raft traffic. The heartbeat usually ensures that the other host is still alive and reachable. In this case, the node that cannot receive any responses from the remaining cluster would get a heartbeat timeout after a couple of seconds, and disconnect from the remaining servers. This is turn should limit further damage and allow the Raft cluster to proceed. Please let me know if I'm missing something else. Edit: There might still be some problems with non-transitive connectivity, where one node can talk to only parts of the cluster. We have built in some protection against this, but don't always handle that case well.