66 ms·

RethinkDB changefeeds are great for dealing with data streams which are common to all users, but changefeeds can get tricky when you need to deal with hundreds

RethinkDB changefeeds are great for dealing with data streams which are common to all users, but changefeeds can get tricky when you need to deal with hundreds of thousands of data streams which are more or less unique to each user (or unique to many small groups of users) - This is particularly true when you need to scale your app beyond a single process/instance/machine. The approach described in this article may not be suitable for a lot of use cases. It's fine if all users shared the same central collection of Fruits (E.g. a giant Fruit basket)... But what if users were divided into many small groups and each group had its own independent Fruit basket? It becomes extremely difficult to manage all these unique changefeeds on the backend (especially if you have to scale beyond a single machine). The team behind RethinkDB is currently working on a project called Horizon (previously called Fusion) which should solve this problem.

kjstevo 11y ago
What approach would you take to solve the problem you describe of individual fruit baskets that can scale?
- jondubois 11y ago
  The simplest way to implement this is using a client-side pub/sub engine and have a channel subscription for each resource on the frontend. So for example, if you have a Fruit basket with id 123 and you want to get realtime updates when that basket changes (E.g. new fruits are added or whatever), you could subscribe to a channel 'fruitbasket/123'. You would have a separate channel for every basket but because the backend treats all channels in a generic way, the complexity on the backend remains minimal (and is handled by your pub/sub engine anyway). You need to make sure that your pub/sub engine is good at handling lots of distinct channels (and is designed to be user-facing - A lot of pubsub engines like Redis are only designed for backend use). There are user-facing pub/sub SaaS services which you can hook into your app but they are a bit expensive. If you want a self-hosted user-facing pub/sub solution, there are only two popular options: - SocketCluster https://github.com/socketcluster/socketcluster https://github.com/socketcluster/socketcluster - Faye https://github.com/faye/faye https://github.com/faye/faye Both Faye and SocketCluster are good at handling lots of unique channels - Based on benchmarks I ran, both can handle the same load of about 10k new subscriptions per second (per CPU core). SC makes scaling across multiple CPU cores trivial so if you have an 8-core machine, you can handle 80K new subscriptions per second with SC out of the box. You can also scale with Faye but it will require a lot more work to get working. Disclosure: I'm the main author of SocketCluster.

kjstevo 11y ago

What approach would you take to solve the problem you describe of individual fruit baskets that can scale?

jondubois 11y ago
The simplest way to implement this is using a client-side pub/sub engine and have a channel subscription for each resource on the frontend. So for example, if you have a Fruit basket with id 123 and you want to get realtime updates when that basket changes (E.g. new fruits are added or whatever), you could subscribe to a channel 'fruitbasket/123'. You would have a separate channel for every basket but because the backend treats all channels in a generic way, the complexity on the backend remains minimal (and is handled by your pub/sub engine anyway). You need to make sure that your pub/sub engine is good at handling lots of distinct channels (and is designed to be user-facing - A lot of pubsub engines like Redis are only designed for backend use). There are user-facing pub/sub SaaS services which you can hook into your app but they are a bit expensive. If you want a self-hosted user-facing pub/sub solution, there are only two popular options: - SocketCluster https://github.com/socketcluster/socketcluster https://github.com/socketcluster/socketcluster - Faye https://github.com/faye/faye https://github.com/faye/faye Both Faye and SocketCluster are good at handling lots of unique channels - Based on benchmarks I ran, both can handle the same load of about 10k new subscriptions per second (per CPU core). SC makes scaling across multiple CPU cores trivial so if you have an 8-core machine, you can handle 80K new subscriptions per second with SC out of the box. You can also scale with Faye but it will require a lot more work to get working. Disclosure: I'm the main author of SocketCluster.