4 ms·

Hey guys, Slava @ RethinkDB here. I'll be around all day to answer questions. We're super-excited about this beta. We've been working on Raft, automatic failov

Hey guys, Slava @ RethinkDB here. I'll be around all day to answer questions. We're super-excited about this beta. We've been working on Raft, automatic failover, and all the relevant enhancements for over a year. It's a massive engineering project that's getting very close to production quality, and it feels so good to finally get it into people's (proverbial) hands!

karlgrz 11y ago
This is so timely, I am really excited to get this new version out on our nodes to kick the tires!
scott_karana 11y ago
Congratulations! Have you been testing this with Jepsen internally, out of curiosity? :)
- coffeemug 11y ago
  Yes, of course! We've also been working with Kyle to understand the Jepsen codebase better. RethinkDB does really well on the Jepsen tests in our internal testing. Of course it's up to Kyle to have the final word :)
fweespeech 11y ago
Are you aiming for production quality over WAN or LAN? Obviously latency is an issue but I'm not sure with the way RethinkDB replicates what the performance penalty would be for WAN in terms of write latency.
- coffeemug 11y ago
  We are aiming for production quality WAN deployments. Many of our customers run RethinkDB in the cloud, and the latency between cloud regions can be quite punishing. We'll document this setup better, but the replication options allow you to safely specify the set of nodes that need sync and async replication (so you can have sync replication for closely connected nodes in a datacenter, and async replication across datacenters).
  - fweespeech 11y ago
    Thanks for the response. :)
lemevi 11y ago
Do you have any plans for windows support for your server?
- coffeemug 11y ago
  Yes, coming soon. See https://github.com/rethinkdb/rethinkdb/issues/1100 https://github.com/rethinkdb/rethinkdb/issues/1100 to track development progress.
  - v3ss0n 11y ago
    Awesomeness , Confirmed.
chrisfosterelli 11y ago
Hi Slava. Excuse me if this is a beginner's question, but on your FAQ[0] page the author argues that it makes more sense for the primary to not failover because this avoids split-brain problems during netsplits. If you've switched to this new Raft algorithm, does this address the split-brain problem? What will now happen if a netsplit occurs and two primaries are elected, then rejoined? [0] http://rethinkdb.com/docs/architecture/#cap-theorem http://rethinkdb.com/docs/architecture/#cap-theorem
- coffeemug 11y ago
  Chris, I think we didn't communicate this well. When we were first building RethinkDB we looked at other similar products and noticed that they don't perform well in split-brain scenarios, and allow all kinds of errors and issues to creep in. For examples of this, just read Aphyr's posts that review various distributed systems. So we thought, first, let's build a system based on manual failover, make that really solid, learn as much as possible about real-world use cases, and then build automatic failover on top of that. This process took a couple of years, and we're finally at a point where we can ship automatic failover that we're confident will behave correctly in real-world (and theoretical) scenarios (modulo bugs that may not have been uncovered in testing). To answer your specific question, if a netsplit occurs, two primaries could never be elected. There will be a primary either one side, or the other side of the split, so rejoining will not be a problem. In practice, however, this is more subtle -- what if there is a netsplit, one side maintains its primary, and the other side elects a new one? What if the user writes to both primaries on either side? What happens after the cluster rejoins? In this specific case, we solve this by requiring the majority of the replicas to acknowledge writes by default before the write acknowledgement is sent to the client. These questions can get really, really subtle -- (for example, what happens if there are multiple cascading netsplits in your cluster?) We wanted to take the time to understand these problems much better before we build automated failover, so we went with a manual failover system first.
  - chrisfosterelli 11y ago
    > In practice, however, this is more subtle -- what if there is a netsplit, one side maintains its primary, and the other side elects a new one? What if the user writes to both primaries on either side? What happens after the cluster rejoins? In this specific case, we solve this by requiring the majority of the replicas to acknowledge writes by default before the write acknowledgement is sent to the client. That's the particular case that I was curious about, that solution makes a lot of sense. I suppose that would also mean that if an outage takes out more than half of your replicas then you will loose all write-abilities (unless I'm mistaken), but that's probably better than the potential mess that could ensue. Do these write requests to nodes that have no route to a primary simply "hang", or are they rejected by the daemon?
mononcqc 11y ago
Have you dealt with mitigating Asymmetric partitions in Raft? http://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-857.pdf http://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-857.pdf shows that a single defective node has the potential to entirely render a Raft cluster useless, specifically in the case of asymmetric partitions: > The mobile devices will often be disconnected from the majority of other devices, thus running election after election without being able to gain enough votes. When these devices rejoin the network, they will force a new election as their term will be much higher than the leader. This leads to frequently disconnected devices becoming the leader when reattached to the network, thus causing regular unavailability of the system when they leave again. > [...] > It is also interesting to consider the impact of a single node being behind an asymmetric partition [...] This node will repeatedly re-run elections as it will not be able to receive incoming messages, each RequestVotes will have a higher term than the last, forcing the leader to step down. Specifically, they required to add client request caches, exposing serial transaction IDs in every single record and log entry, specific timeouts and measures for livelock bugs this introduced, which include replacing an entry by a noop value to be replicated on some read-only operation done at a later time. On top of this, these modifications may end up reducing the serializability constructs to bring it closer to what a Master-Follower replication may give in a DB -- i.e. it doesn't guarantee one true serializable sequence of commits anymore, but only that the state replication will be consistent across all nodes
- danielmewes 11y ago
  We haven't added special logic for handling such cases. I think having servers with particularly unstable connectivity in a RethinkDB cluster is really uncommon though. If you want to run a RethinkDB server on your mobile device that's a different scenario of course...
  - mononcqc 11y ago
    Well the problem is obvious in mobile devices, but any asymmetric netsplit will result in a single node taking down your cluster for as long as the netsplit lasts.

karlgrz 11y ago

This is so timely, I am really excited to get this new version out on our nodes to kick the tires!

scott_karana 11y ago

Congratulations! Have you been testing this with Jepsen internally, out of curiosity? :)

coffeemug 11y ago
Yes, of course! We've also been working with Kyle to understand the Jepsen codebase better. RethinkDB does really well on the Jepsen tests in our internal testing. Of course it's up to Kyle to have the final word :)

fweespeech 11y ago

Are you aiming for production quality over WAN or LAN? Obviously latency is an issue but I'm not sure with the way RethinkDB replicates what the performance penalty would be for WAN in terms of write latency.

coffeemug 11y ago
We are aiming for production quality WAN deployments. Many of our customers run RethinkDB in the cloud, and the latency between cloud regions can be quite punishing. We'll document this setup better, but the replication options allow you to safely specify the set of nodes that need sync and async replication (so you can have sync replication for closely connected nodes in a datacenter, and async replication across datacenters).
- fweespeech 11y ago
  Thanks for the response. :)

lemevi 11y ago

Do you have any plans for windows support for your server?

coffeemug 11y ago
Yes, coming soon. See https://github.com/rethinkdb/rethinkdb/issues/1100 https://github.com/rethinkdb/rethinkdb/issues/1100 to track development progress.
- v3ss0n 11y ago
  Awesomeness , Confirmed.

chrisfosterelli 11y ago

Hi Slava. Excuse me if this is a beginner's question, but on your FAQ[0] page the author argues that it makes more sense for the primary to not failover because this avoids split-brain problems during netsplits. If you've switched to this new Raft algorithm, does this address the split-brain problem? What will now happen if a netsplit occurs and two primaries are elected, then rejoined? [0] http://rethinkdb.com/docs/architecture/#cap-theorem http://rethinkdb.com/docs/architecture/#cap-theorem

coffeemug 11y ago
Chris, I think we didn't communicate this well. When we were first building RethinkDB we looked at other similar products and noticed that they don't perform well in split-brain scenarios, and allow all kinds of errors and issues to creep in. For examples of this, just read Aphyr's posts that review various distributed systems. So we thought, first, let's build a system based on manual failover, make that really solid, learn as much as possible about real-world use cases, and then build automatic failover on top of that. This process took a couple of years, and we're finally at a point where we can ship automatic failover that we're confident will behave correctly in real-world (and theoretical) scenarios (modulo bugs that may not have been uncovered in testing). To answer your specific question, if a netsplit occurs, two primaries could never be elected. There will be a primary either one side, or the other side of the split, so rejoining will not be a problem. In practice, however, this is more subtle -- what if there is a netsplit, one side maintains its primary, and the other side elects a new one? What if the user writes to both primaries on either side? What happens after the cluster rejoins? In this specific case, we solve this by requiring the majority of the replicas to acknowledge writes by default before the write acknowledgement is sent to the client. These questions can get really, really subtle -- (for example, what happens if there are multiple cascading netsplits in your cluster?) We wanted to take the time to understand these problems much better before we build automated failover, so we went with a manual failover system first.
- chrisfosterelli 11y ago
  > In practice, however, this is more subtle -- what if there is a netsplit, one side maintains its primary, and the other side elects a new one? What if the user writes to both primaries on either side? What happens after the cluster rejoins? In this specific case, we solve this by requiring the majority of the replicas to acknowledge writes by default before the write acknowledgement is sent to the client. That's the particular case that I was curious about, that solution makes a lot of sense. I suppose that would also mean that if an outage takes out more than half of your replicas then you will loose all write-abilities (unless I'm mistaken), but that's probably better than the potential mess that could ensue. Do these write requests to nodes that have no route to a primary simply "hang", or are they rejected by the daemon?

mononcqc 11y ago

Have you dealt with mitigating Asymmetric partitions in Raft? http://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-857.pdf http://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-857.pdf shows that a single defective node has the potential to entirely render a Raft cluster useless, specifically in the case of asymmetric partitions: > The mobile devices will often be disconnected from the majority of other devices, thus running election after election without being able to gain enough votes. When these devices rejoin the network, they will force a new election as their term will be much higher than the leader. This leads to frequently disconnected devices becoming the leader when reattached to the network, thus causing regular unavailability of the system when they leave again. > [...] > It is also interesting to consider the impact of a single node being behind an asymmetric partition [...] This node will repeatedly re-run elections as it will not be able to receive incoming messages, each RequestVotes will have a higher term than the last, forcing the leader to step down. Specifically, they required to add client request caches, exposing serial transaction IDs in every single record and log entry, specific timeouts and measures for livelock bugs this introduced, which include replacing an entry by a noop value to be replicated on some read-only operation done at a later time. On top of this, these modifications may end up reducing the serializability constructs to bring it closer to what a Master-Follower replication may give in a DB -- i.e. it doesn't guarantee one true serializable sequence of commits anymore, but only that the state replication will be consistent across all nodes

danielmewes 11y ago
We haven't added special logic for handling such cases. I think having servers with particularly unstable connectivity in a RethinkDB cluster is really uncommon though. If you want to run a RethinkDB server on your mobile device that's a different scenario of course...
- mononcqc 11y ago
  Well the problem is obvious in mobile devices, but any asymmetric netsplit will result in a single node taking down your cluster for as long as the netsplit lasts.