11 ms·

RethinkDB 2.1 beta: automatic failover with Raft

135 points by coffeemug 11y ago

thelinuxl1ch 11y ago
Being developing with RethinkDB for one year, I can say that it's amazing how fast this database evolved, the API is very flexible, the developer team cares a lot about user feedback, it's a thriving community! Auto failover will be a life saver, but I'm much more anxious for reliable changefeeds :)
- danielmewes 11y ago
  (Daniel at RethinkDB here) Thanks for the nice words. We're going to add changefeeds that can survive short client disconnects in 2.2, and add further reliability features for changefeeds in the following versions.
- coffeemug 11y ago
  Reliable feeds (and much more) is coming in 2.2. I think we found an elegant solution[1] that will make different types of users happy, so it'll be exciting to ship it! [1] https://github.com/rethinkdb/rethinkdb/issues/3471 https://github.com/rethinkdb/rethinkdb/issues/3471
  - thelinuxl1ch 11y ago
    Nice, this will open a whole world of use cases
  - v3ss0n 11y ago
    Thats very nice to hear, Can't wait for Reliable change feeds.
coffeemug 11y ago
Hey guys, Slava @ RethinkDB here. I'll be around all day to answer questions. We're super-excited about this beta. We've been working on Raft, automatic failover, and all the relevant enhancements for over a year. It's a massive engineering project that's getting very close to production quality, and it feels so good to finally get it into people's (proverbial) hands!
- karlgrz 11y ago
  This is so timely, I am really excited to get this new version out on our nodes to kick the tires!
- scott_karana 11y ago
  Congratulations! Have you been testing this with Jepsen internally, out of curiosity? :)
  - coffeemug 11y ago
    Yes, of course! We've also been working with Kyle to understand the Jepsen codebase better. RethinkDB does really well on the Jepsen tests in our internal testing. Of course it's up to Kyle to have the final word :)
- fweespeech 11y ago
  Are you aiming for production quality over WAN or LAN? Obviously latency is an issue but I'm not sure with the way RethinkDB replicates what the performance penalty would be for WAN in terms of write latency.
  - coffeemug 11y ago
    We are aiming for production quality WAN deployments. Many of our customers run RethinkDB in the cloud, and the latency between cloud regions can be quite punishing. We'll document this setup better, but the replication options allow you to safely specify the set of nodes that need sync and async replication (so you can have sync replication for closely connected nodes in a datacenter, and async replication across datacenters).
    - fweespeech 11y ago
      Thanks for the response. :)
- lemevi 11y ago
  Do you have any plans for windows support for your server?
thelinuxl1ch 11y ago
Are there any performance issues we should know after the Raft implementation?
- danielmewes 11y ago
  General query performance should be the same, if not better in this release (the improvements are not due to Raft, but due to other changes). We use Raft only to determine a consistent configuration. The queries themselves are executed in the same efficient way as before. Note that in the beta, reconfiguring tables over servers on rotational disks can be slow https://github.com/rethinkdb/rethinkdb/issues/4279 https://github.com/rethinkdb/rethinkdb/issues/4279 . If you store large documents, you might also see some increases in memory usage during backfills https://github.com/rethinkdb/rethinkdb/issues/4474 https://github.com/rethinkdb/rethinkdb/issues/4474 . We're going to address both of these issues before the final release.
  - DanWaterworth 11y ago
    > We use Raft only to determine a consistent configuration. The queries themselves are executed in the same efficient way as before. So, you've made the noob mistake in using Raft, where only configuration goes in the log and actual changes to data aren't replicated in the same manner.
    - danielmewes 11y ago
      We have carefully designed the way data is versioned and replicated in RethinkDB 2.1 to result in a correct and consistent system in combination with the Raft-supported configuration management. For example we make sure that both components use consistent quorum definitions and the same membership information at any point. This allows us to provide different degrees of consistency guarantees for the data, depending on the user's need. Our default is already pretty conservative, but allows uncommitted data to be visible by reads. In the strongest consistency mode, RethinkDB 2.1 provides full linearizability (see http://docs.rethinkdb.com/2.1/docs/consistency/ http://docs.rethinkdb.com/2.1/docs/consistency/ for details). We have confirmed this both theoretically as well as by testing the overall system using the Jepsen test framework.
RoboTeddy 11y ago
I'm continually impressed with the rate and quality of rethinkdb's advancement. There's some really fantastic software engineering going on over there.
- v3ss0n 11y ago
  The way RethinkDB crews co-operate over Github is fantastic. Every workflow is tru github and you can transparently see their operation. I learnt a lot from that and i apply that to my team. We are building "A Killer" Communication tool using RethinkDB + HTML5. We will be announcing soon.
  - eterps 11y ago
    Can you give an example how their cooperation over Github is special?
    - pests 11y ago
      I really can't give you a concrete answer but I wanted to confirm what my GP said: I must admit to being obsessed with their GitHub Issues for awhile. It was almost as addicting as HN or reddit to check and read new issues and comments - I've probably read every major discussion and minor interaction they've had over the last year or more.
    - v3ss0n 11y ago
      I am with @pest . rethinkdb's github issues becomes my daily read, along with Github's Atom repo. https://github.com/rethinkdb/rethinkdb https://github.com/rethinkdb/rethinkdb Just look at the issues , how they handle them , how their professional responses , and how quick they resolves issues after issues with full co-operation between issue owner , users and contributor. They guide contributors as well. And it is really Addictive , be warned.
- carterehsmith 11y ago
  Quite impressive, and very honest, too. They provide a "When is RethinkDB not a good choice?" point, they provide a "stability report", and that report was a proper torture test, and they responded promptly to the issues raised by the Jepsen test. BTW I have no use for their db at the time, but if I did one day, I would pick RethinkDB in a heartbeat over all competitors.
ascendantlogic 11y ago
Every time I say to myself "How can Rethink get more awesome?" I am shown how. What a great product.
- v3ss0n 11y ago
  well said.
nodesocket 11y ago
@cofeemug, can you describe a typical HA deployments? Are 3 nodes required to start? If you want to shard across 3 nodes, but also sustain a 1 node failure, how many nodes are required? Finally, how about shard across 3 nodes, but sustain a two node failure?
- danielmewes 11y ago
  3 nodes are indeed required to have automatic failover working. We recommend using 3 (or more) nodes and replicating all tables with a replication factor of 3. That way each node is going to maintain a full copy of the data. In case of a 1 node failure (with or without sharding, as long as the replication factor is set to 3), one of the two remaining servers is going to take over for the primary roles that might have been hosted on the missing server. If you want to sustain a two node failure without any data loss, you will need 5 servers and also set the replication factor to 5. 5 is the lowest number that guarantees that if two nodes fail, you will still be left with a majority of replicas (i.e. 3 out of 5). A majority like that is required to guarantee data durability and to enable auto failover. If you are ok with losing a few seconds worth of changes and do not require automatic failover, even a 3 node setup can be enough to sustain a two node failure. In that case you will have to perform a manual "emergency_repair" to recover availability (see http://docs.rethinkdb.com/2.1/api/javascript/reconfigure/ http://docs.rethinkdb.com/2.1/api/javascript/reconfigure/ for details), but most of the data should still be there. In addition you can shard the table into 3 shards for additional performance gains. This is for the most part unrelated to availability and data safety.
  - thelinuxl1ch 11y ago
    So, on a minimum three-node setup, three replicas + three shards per table are doable? Are there any caveats?
    - coffeemug 11y ago
      Yes, doable and desirable. There shouldn't be any caveats to this deployment.
  - 11y ago
v3ss0n 11y ago
I havn't check the PRs yet , but does this BETA also have performance improvement commit a few weeks ago? https://github.com/rethinkdb/rethinkdb/issues/4441 https://github.com/rethinkdb/rethinkdb/issues/4441 That one rethinkdb overall performance of writes by 2x right?
- danielmewes 11y ago
  Yes, that patch for soft durability writes is in there. Looking forward to hear if it improves things for you.
  - v3ss0n 11y ago
    thanks , i am going to test after building.
jtwebman 11y ago
I have played around with this database I am looking forward to using it on my next project! This is great news but I would love if they wrote a Erlang/Elixir driver. There is one that that the community wrote but would love to see someone full time on it.
contingencies 11y ago
Don't get me wrong, it's good to see people building alternative database systems, and it's great to see the community as a whole embracing the architectural issues of high availability through a combination of Paxos and Raft. However, I'm not sure that we should be supporting every damn service having its own rolled-in implementation of its own choice of consensus algorithm. Why not limit service scope to service provisioning, and normalize to a reasonable service-independent solution for high availability consensus and orchestration?This leads to far more solid and well tested operations practices that respect service interactions and you don't wind up with "we swept it under the rug" problems like the RethinkDB 2.1 beta announcement's "with only a few seconds of unavailability". Come on guys, call a spade a spade.
- coffeemug 11y ago
  We explored this possibility in quite a bit of depth, but this turns out to be much harder to do in practice than it seems. There are currently two types of projects that offer this kind of service. The first is built in C/C++ and packaged as a library. We tried to use these libraries, but a project of RethinkDB's scope has a lot of abstractions for networking, threading, memory allocation, etc. We quickly found that we couldn't effectively integrate existing libraries into our coroutine system and networking stack. The second type is projects that are typically built in higher level languages and run standalone. We tried that too, but it turns out to be a user experience nightmare. Users would need to know how to configure these different services, and would have to deal with deployment. The configuration aspect is quite difficult -- it's hard to configure these services in the right way without making a mistake (to make them work for the needs of a database system). We also looked into abstracting that away by building a porcelain UI, but there are a lot of challenges there that are very difficult to overcome. Finally, RethinkDB's needs are quite specialized and there isn't a service that does what we need out of the box. We looked at implementing Raft ourselves and realized it isn't actually hard -- a much bigger challenge is properly architecting the rest of the system. To give you an idea of timing, it took us two weeks to get a Raft implementation, and another few weeks to polish it. It took another year to get everything else working (plus two years of expertise in the field wrt real-world issues that users encounter). > only a few seconds of unavailability This isn't something you can escape in system that's based on authoritative replicas. All other systems that use this architecture face this problem; an external service wouldn't solve it. TL;DR: it would be really nice if we could use an existing service, but unfortunately this is a much trickier problem than it first appears.
aaronorosen 11y ago
Awesome to see! I tried giving this a shot starting up 3 rethinkdb nodes with a proxy node with my existing application. Setup 3 shards and 2 replicas. Unfortunately, I wasn't able to get my application working. The UI had the tables going back to outdated reads (everything was able to initialize though once). I'll give it another shot next week and report back. I'm sure it could also be a problem with my setup maybe.
- coffeemug 11y ago
  Could you submit a bug report at https://github.com/rethinkdb/rethinkdb/issues/new https://github.com/rethinkdb/rethinkdb/issues/new ? It would help immensely!
shockzzz 11y ago
What's the most mature client driver for RethinkDB, in your opinion?
- coffeemug 11y ago
  Ruby, Python, and JavaScript are all equally mature (and developed in-house full time). Of the community drivers, Go is probably the most mature (but other community drivers, like the .NET one are really nice).

135 points by coffeemug 11y ago

Being developing with RethinkDB for one year, I can say that it's amazing how fast this database evolved, the API is very flexible, the developer team cares a lot about user feedback, it's a thriving community! Auto failover will be a life saver, but I'm much more anxious for reliable changefeeds :)

danielmewes 11y ago
(Daniel at RethinkDB here) Thanks for the nice words. We're going to add changefeeds that can survive short client disconnects in 2.2, and add further reliability features for changefeeds in the following versions.
coffeemug 11y ago
Reliable feeds (and much more) is coming in 2.2. I think we found an elegant solution[1] that will make different types of users happy, so it'll be exciting to ship it! [1] https://github.com/rethinkdb/rethinkdb/issues/3471 https://github.com/rethinkdb/rethinkdb/issues/3471
- thelinuxl1ch 11y ago
  Nice, this will open a whole world of use cases
- v3ss0n 11y ago
  Thats very nice to hear, Can't wait for Reliable change feeds.

coffeemug 11y ago

Hey guys, Slava @ RethinkDB here. I'll be around all day to answer questions. We're super-excited about this beta. We've been working on Raft, automatic failover, and all the relevant enhancements for over a year. It's a massive engineering project that's getting very close to production quality, and it feels so good to finally get it into people's (proverbial) hands!

karlgrz 11y ago
This is so timely, I am really excited to get this new version out on our nodes to kick the tires!
scott_karana 11y ago
Congratulations! Have you been testing this with Jepsen internally, out of curiosity? :)
- coffeemug 11y ago
  Yes, of course! We've also been working with Kyle to understand the Jepsen codebase better. RethinkDB does really well on the Jepsen tests in our internal testing. Of course it's up to Kyle to have the final word :)
fweespeech 11y ago
Are you aiming for production quality over WAN or LAN? Obviously latency is an issue but I'm not sure with the way RethinkDB replicates what the performance penalty would be for WAN in terms of write latency.
- coffeemug 11y ago
  We are aiming for production quality WAN deployments. Many of our customers run RethinkDB in the cloud, and the latency between cloud regions can be quite punishing. We'll document this setup better, but the replication options allow you to safely specify the set of nodes that need sync and async replication (so you can have sync replication for closely connected nodes in a datacenter, and async replication across datacenters).
  - fweespeech 11y ago
    Thanks for the response. :)
lemevi 11y ago
Do you have any plans for windows support for your server?

thelinuxl1ch 11y ago

Are there any performance issues we should know after the Raft implementation?

danielmewes 11y ago
General query performance should be the same, if not better in this release (the improvements are not due to Raft, but due to other changes). We use Raft only to determine a consistent configuration. The queries themselves are executed in the same efficient way as before. Note that in the beta, reconfiguring tables over servers on rotational disks can be slow https://github.com/rethinkdb/rethinkdb/issues/4279 https://github.com/rethinkdb/rethinkdb/issues/4279 . If you store large documents, you might also see some increases in memory usage during backfills https://github.com/rethinkdb/rethinkdb/issues/4474 https://github.com/rethinkdb/rethinkdb/issues/4474 . We're going to address both of these issues before the final release.
- DanWaterworth 11y ago
  > We use Raft only to determine a consistent configuration. The queries themselves are executed in the same efficient way as before. So, you've made the noob mistake in using Raft, where only configuration goes in the log and actual changes to data aren't replicated in the same manner.
  - danielmewes 11y ago
    We have carefully designed the way data is versioned and replicated in RethinkDB 2.1 to result in a correct and consistent system in combination with the Raft-supported configuration management. For example we make sure that both components use consistent quorum definitions and the same membership information at any point. This allows us to provide different degrees of consistency guarantees for the data, depending on the user's need. Our default is already pretty conservative, but allows uncommitted data to be visible by reads. In the strongest consistency mode, RethinkDB 2.1 provides full linearizability (see http://docs.rethinkdb.com/2.1/docs/consistency/ http://docs.rethinkdb.com/2.1/docs/consistency/ for details). We have confirmed this both theoretically as well as by testing the overall system using the Jepsen test framework.

RoboTeddy 11y ago

I'm continually impressed with the rate and quality of rethinkdb's advancement. There's some really fantastic software engineering going on over there.

v3ss0n 11y ago
The way RethinkDB crews co-operate over Github is fantastic. Every workflow is tru github and you can transparently see their operation. I learnt a lot from that and i apply that to my team. We are building "A Killer" Communication tool using RethinkDB + HTML5. We will be announcing soon.
- eterps 11y ago
  Can you give an example how their cooperation over Github is special?
  - pests 11y ago
    I really can't give you a concrete answer but I wanted to confirm what my GP said: I must admit to being obsessed with their GitHub Issues for awhile. It was almost as addicting as HN or reddit to check and read new issues and comments - I've probably read every major discussion and minor interaction they've had over the last year or more.
  - v3ss0n 11y ago
    I am with @pest . rethinkdb's github issues becomes my daily read, along with Github's Atom repo. https://github.com/rethinkdb/rethinkdb https://github.com/rethinkdb/rethinkdb Just look at the issues , how they handle them , how their professional responses , and how quick they resolves issues after issues with full co-operation between issue owner , users and contributor. They guide contributors as well. And it is really Addictive , be warned.
carterehsmith 11y ago
Quite impressive, and very honest, too. They provide a "When is RethinkDB not a good choice?" point, they provide a "stability report", and that report was a proper torture test, and they responded promptly to the issues raised by the Jepsen test. BTW I have no use for their db at the time, but if I did one day, I would pick RethinkDB in a heartbeat over all competitors.

ascendantlogic 11y ago

Every time I say to myself "How can Rethink get more awesome?" I am shown how. What a great product.

v3ss0n 11y ago
well said.

nodesocket 11y ago

@cofeemug, can you describe a typical HA deployments? Are 3 nodes required to start? If you want to shard across 3 nodes, but also sustain a 1 node failure, how many nodes are required? Finally, how about shard across 3 nodes, but sustain a two node failure?

danielmewes 11y ago
3 nodes are indeed required to have automatic failover working. We recommend using 3 (or more) nodes and replicating all tables with a replication factor of 3. That way each node is going to maintain a full copy of the data. In case of a 1 node failure (with or without sharding, as long as the replication factor is set to 3), one of the two remaining servers is going to take over for the primary roles that might have been hosted on the missing server. If you want to sustain a two node failure without any data loss, you will need 5 servers and also set the replication factor to 5. 5 is the lowest number that guarantees that if two nodes fail, you will still be left with a majority of replicas (i.e. 3 out of 5). A majority like that is required to guarantee data durability and to enable auto failover. If you are ok with losing a few seconds worth of changes and do not require automatic failover, even a 3 node setup can be enough to sustain a two node failure. In that case you will have to perform a manual "emergency_repair" to recover availability (see http://docs.rethinkdb.com/2.1/api/javascript/reconfigure/ http://docs.rethinkdb.com/2.1/api/javascript/reconfigure/ for details), but most of the data should still be there. In addition you can shard the table into 3 shards for additional performance gains. This is for the most part unrelated to availability and data safety.
- thelinuxl1ch 11y ago
  So, on a minimum three-node setup, three replicas + three shards per table are doable? Are there any caveats?
  - coffeemug 11y ago
    Yes, doable and desirable. There shouldn't be any caveats to this deployment.
- 11y ago

v3ss0n 11y ago

I havn't check the PRs yet , but does this BETA also have performance improvement commit a few weeks ago? https://github.com/rethinkdb/rethinkdb/issues/4441 https://github.com/rethinkdb/rethinkdb/issues/4441 That one rethinkdb overall performance of writes by 2x right?

danielmewes 11y ago
Yes, that patch for soft durability writes is in there. Looking forward to hear if it improves things for you.
- v3ss0n 11y ago
  thanks , i am going to test after building.

jtwebman 11y ago

I have played around with this database I am looking forward to using it on my next project! This is great news but I would love if they wrote a Erlang/Elixir driver. There is one that that the community wrote but would love to see someone full time on it.

contingencies 11y ago

Don't get me wrong, it's good to see people building alternative database systems, and it's great to see the community as a whole embracing the architectural issues of high availability through a combination of Paxos and Raft. However, I'm not sure that we should be supporting every damn service having its own rolled-in implementation of its own choice of consensus algorithm. Why not limit service scope to service provisioning, and normalize to a reasonable service-independent solution for high availability consensus and orchestration?This leads to far more solid and well tested operations practices that respect service interactions and you don't wind up with "we swept it under the rug" problems like the RethinkDB 2.1 beta announcement's "with only a few seconds of unavailability". Come on guys, call a spade a spade.

coffeemug 11y ago
We explored this possibility in quite a bit of depth, but this turns out to be much harder to do in practice than it seems. There are currently two types of projects that offer this kind of service. The first is built in C/C++ and packaged as a library. We tried to use these libraries, but a project of RethinkDB's scope has a lot of abstractions for networking, threading, memory allocation, etc. We quickly found that we couldn't effectively integrate existing libraries into our coroutine system and networking stack. The second type is projects that are typically built in higher level languages and run standalone. We tried that too, but it turns out to be a user experience nightmare. Users would need to know how to configure these different services, and would have to deal with deployment. The configuration aspect is quite difficult -- it's hard to configure these services in the right way without making a mistake (to make them work for the needs of a database system). We also looked into abstracting that away by building a porcelain UI, but there are a lot of challenges there that are very difficult to overcome. Finally, RethinkDB's needs are quite specialized and there isn't a service that does what we need out of the box. We looked at implementing Raft ourselves and realized it isn't actually hard -- a much bigger challenge is properly architecting the rest of the system. To give you an idea of timing, it took us two weeks to get a Raft implementation, and another few weeks to polish it. It took another year to get everything else working (plus two years of expertise in the field wrt real-world issues that users encounter). > only a few seconds of unavailability This isn't something you can escape in system that's based on authoritative replicas. All other systems that use this architecture face this problem; an external service wouldn't solve it. TL;DR: it would be really nice if we could use an existing service, but unfortunately this is a much trickier problem than it first appears.

aaronorosen 11y ago

Awesome to see! I tried giving this a shot starting up 3 rethinkdb nodes with a proxy node with my existing application. Setup 3 shards and 2 replicas. Unfortunately, I wasn't able to get my application working. The UI had the tables going back to outdated reads (everything was able to initialize though once). I'll give it another shot next week and report back. I'm sure it could also be a problem with my setup maybe.

coffeemug 11y ago
Could you submit a bug report at https://github.com/rethinkdb/rethinkdb/issues/new https://github.com/rethinkdb/rethinkdb/issues/new ? It would help immensely!

shockzzz 11y ago

What's the most mature client driver for RethinkDB, in your opinion?

coffeemug 11y ago
Ruby, Python, and JavaScript are all equally mature (and developed in-house full time). Of the community drivers, Go is probably the most mature (but other community drivers, like the .NET one are really nice).