4 ms·

Interesting to see RethinkDB being re-posted on here again, even if it is a 3 year old article. We use it extensively in our background worker processing of ou

Interesting to see RethinkDB being re-posted on here again, even if it is a 3 year old article. We use it extensively in our background worker processing of our SaaS and it has worked magnificently for years. I even write a small HN related fun project with it a couple of years back [0]. I haven't kept up with how it is travelling as an open source project, but I am hoping that community interest is still strong and that the project will be around for a while yet. [0]- https://hackernoon.com/tophn-a-fun-side-project-built-with-vue-js-and-rethinkdb-f22159641c1b https://hackernoon.com/tophn-a-fun-side-project-built-with-v...

drdaeman 7y ago
> it has worked magnificently for years. It surely did the job, but unfortunately, I have different experience. - It's very space-ineffective. One particular thing I've discovered is that it's very ineffective for storing numbers. A strings are much more space-efficient (https://github.com/rethinkdb/rethinkdb/issues/6304 https://github.com/rethinkdb/rethinkdb/issues/6304). However, there must be something else. I'm slowly moving to PostgreSQL (experimenting with CockroachDB as alternative) and maybe that's just me, but similar datasets with similar indices require significantly less disk space there. - It dies under heavy load. That could be a tolerable issue - I can throttle updates my app generate, but there's no way to control replication speed. Trying to add replicas to some of larger tables reliably kills the destination node for me. - There's no way to limit memory, and optimizing for this is hard. A node with 64GiB RAM lives okay, but the one with only 16GiB gets periodically OOM-killed (cache size on that node is set to mere 6GiB). - Sometimes, it has some weird cluster discovery delays, where restarted nodes just sit and don't connect to their peers for minutes, even though they're reachable and there are no network issues. This is not a problem for large clusters, but I have only 3 nodes (cost constraints), where a loss of one moves system into partially degraded state as replicating all tables is too wasteful. Tolerable, but not fun. > how it is travelling as an open source project Unfortunately, I'd say it is dead. There is some discussion activity, but almost no code activity.
- cryptica 7y ago
  I'm launching a scalable open source cryptocurrency payment system using RethinkDB. It's amazing how well it works and scales on Kubernetes at the click of a button. There is a video of it on my project website https://crypticle.io/ https://crypticle.io/ I intend to keep using it for as many future projects as possible. AFAIK, the business did not survive because the product was too good and didn't have enough problems to charge support for. Actually this is a big problem in open source and that's why I'm involved in cryptocurrency now.
  - drdaeman 7y ago
    > It's amazing how well it works and scales on Kubernetes at the click of a button Oh, just try to scale it down ;) https://github.com/rethinkdb/docs/issues/958 https://github.com/rethinkdb/docs/issues/958 Deleting from system table is okay, but messing with server tags (when the setup is heterogeneous) is not exactly fun. To be fair, that's a very minor issue. Perfectly manageable, just a slight inconvenience when you figure things out.
    - cryptica 7y ago
      It's not that difficult, you can use the reconfigure query to put all the shards on one server, then you can scale down the other servers.
l33tman 7y ago
We've used it for years as well, and (for us) the absolutely worst fact about it is that it is a memory hog like crazy. Even without any accesses at all, it needs like 32 GB RAM on our nodes, for a 0.5 TB DB (with a bunch of indexes). I guess this is by design and is a compromise that requires this, but it makes it very expensive to deploy on AWS, where RAM is the real differentiator for this (actually storing the table on EBS is super cheap compared to the RAM required...). It's also very difficult to analyse to see where the RAM is going, this is one thing at least that could be improved. I would not use it for any newer projects unless this was fixed.

drdaeman 7y ago

> it has worked magnificently for years. It surely did the job, but unfortunately, I have different experience. - It's very space-ineffective. One particular thing I've discovered is that it's very ineffective for storing numbers. A strings are much more space-efficient (https://github.com/rethinkdb/rethinkdb/issues/6304 https://github.com/rethinkdb/rethinkdb/issues/6304). However, there must be something else. I'm slowly moving to PostgreSQL (experimenting with CockroachDB as alternative) and maybe that's just me, but similar datasets with similar indices require significantly less disk space there. - It dies under heavy load. That could be a tolerable issue - I can throttle updates my app generate, but there's no way to control replication speed. Trying to add replicas to some of larger tables reliably kills the destination node for me. - There's no way to limit memory, and optimizing for this is hard. A node with 64GiB RAM lives okay, but the one with only 16GiB gets periodically OOM-killed (cache size on that node is set to mere 6GiB). - Sometimes, it has some weird cluster discovery delays, where restarted nodes just sit and don't connect to their peers for minutes, even though they're reachable and there are no network issues. This is not a problem for large clusters, but I have only 3 nodes (cost constraints), where a loss of one moves system into partially degraded state as replicating all tables is too wasteful. Tolerable, but not fun. > how it is travelling as an open source project Unfortunately, I'd say it is dead. There is some discussion activity, but almost no code activity.

cryptica 7y ago
I'm launching a scalable open source cryptocurrency payment system using RethinkDB. It's amazing how well it works and scales on Kubernetes at the click of a button. There is a video of it on my project website https://crypticle.io/ https://crypticle.io/ I intend to keep using it for as many future projects as possible. AFAIK, the business did not survive because the product was too good and didn't have enough problems to charge support for. Actually this is a big problem in open source and that's why I'm involved in cryptocurrency now.
- drdaeman 7y ago
  > It's amazing how well it works and scales on Kubernetes at the click of a button Oh, just try to scale it down ;) https://github.com/rethinkdb/docs/issues/958 https://github.com/rethinkdb/docs/issues/958 Deleting from system table is okay, but messing with server tags (when the setup is heterogeneous) is not exactly fun. To be fair, that's a very minor issue. Perfectly manageable, just a slight inconvenience when you figure things out.
  - cryptica 7y ago
    It's not that difficult, you can use the reconfigure query to put all the shards on one server, then you can scale down the other servers.

l33tman 7y ago

We've used it for years as well, and (for us) the absolutely worst fact about it is that it is a memory hog like crazy. Even without any accesses at all, it needs like 32 GB RAM on our nodes, for a 0.5 TB DB (with a bunch of indexes). I guess this is by design and is a compromise that requires this, but it makes it very expensive to deploy on AWS, where RAM is the real differentiator for this (actually storing the table on EBS is super cheap compared to the RAM required...). It's also very difficult to analyse to see where the RAM is going, this is one thing at least that could be improved. I would not use it for any newer projects unless this was fixed.