5 ms·

I was looking really hard at using CouchDB 2.0 for an offline-first progressive web app. The biggest asset in my opinion of CouchDB is actually PouchDB [1] Po

by tarr11 9y ago

I was looking really hard at using CouchDB 2.0 for an offline-first progressive web app. The biggest asset in my opinion of CouchDB is actually PouchDB [1] PouchDB is really great and nicely designed. However, the showstopper that I hit was that CouchDB still does not support permissions on a per-user basis (even though PouchDB encourages this explicitly [2]) What this means is that if you have a typical system with users and logins, who can only access their own data, and you require some sort of aggregated view where you can view all users data behind some sort of permissions (admin user, etc), you are forced to create a new db for each user in Couch, and then run replication for those users to a separate master. Think for example if you wanted to store images in CouchDB, and then have some sort of global feed (like instagram) So, systems with many users (tens of thousands -> millions) would be stuck replicating an equal number of couch databases constantly. Each replica in 2.0 was an erlang thread, and each database is a separate file. Apparently some of the performance issues were fixed in 2.1, but it still requires a physical file per db and continuous replication. Some promising solutions like envoy [3] from Cloudant, which were based on Mango [4] have been worked on, but don't seem to have much support in the core Couch community and feels to have been abandoned. Another problem is that you can't really throttle or manage pouchDB <-> couchDB replication very well. Eg, if you have a large-ish database that's out of sync, it's just going to try to download itself as soon as you sync, which isn't very mobile friendly, or "PWA" friendly IMO. Net result for me was that I just stuck with my Postgres DB, Rails API, and a bunch of Redux/React code to accomplish what I needed. Still a fan of Couch/Pouch but wouldn't recommend it if your system requires any kind of serious per-user or role-based permissions or aggregated views. [1] https://pouchdb.com/ https://pouchdb.com/ [2] https://pouchdb.com/2015/04/05/filtered-replication.html https://pouchdb.com/2015/04/05/filtered-replication.html [3] https://github.com/cloudant-labs/envoy https://github.com/cloudant-labs/envoy [4] https://blog.couchdb.org/2016/08/03/feature-mango-query/ https://blog.couchdb.org/2016/08/03/feature-mango-query/

sagelywizard 9y ago
The main feature which was added in 2.1 was the replication scheduler, which makes it possible to run millions of replications concurrently. That makes the database-per-user model much more feasible. As for throttling, the CouchDB replicator gracefully handles HTTP 429, but that wouldn't help with PouchDB-managed replications. I can't really speak on PouchDB, but maybe it's possible to limit the replication on the client side, since that's where it's managed?
tropshop 9y ago
> So, systems with many users (tens of thousands -> millions) would be stuck replicating an equal number of couch databases constantly I also ran into this when using `continuous: true` to create persistent replication, which suffers from this 1-to-1 resource problem. I now use a single listener on `_db_changes` and fire one-off replication to the aggregate db. I'm also keeping an eye on spiegel[1] I'm curious on how you handle offline support, sync conflicts, and multiple devices. I started with CouchDB 1.x and have been through enough growing pains to wonder how things would have been to stick with SQL, but I think the future is only brighter. CouchDB 2.1 looks great, and I hope the entire erlang ecosystem continues to see growth and adoption. [1] https://github.com/redgeoff/spiegel https://github.com/redgeoff/spiegel
- tarr11 9y ago
  I know that it is a common trope that Couch / Pouch seems to think that SQL databases don't work for offline replication and you need their "batteries-included" solution. However, it's not the only solution and every app in the app store that works offline has had to deal with it. Most of those are using SQL databases. In truth, what Couch does is force you to handle conflict events which gives you a convenient platform level place to put some of your automatic conflict/merge code. A lot of merge code is UI however and Couch does not help with that at all. Detecting conflicts is a subtle thing, but it's not necessarily a one-size-fits-all problem as couch would have us believe. I It also gives you versioning for free (which is pretty easy to implement in SQL) However, at the end of the day, every developer must make application level decisions on how to handle merge conflicts. Couch/Pouch does not obviate this - it just forces you to deal with it and lets you think about your application a little differently. Once you have to deal with merges and conflicts in your application code, it's more a matter of designing your application to have fewer conflicts (either through better merging, vector clocks, pessimistic locking, etc) I wish I could have used it but all my other data is in a battle-tested Postgres db, being nicely backed up regularly on Heroku without fail. My CouchDB was on a Google Cloud server, using a brand new replication framework as backup. I don't really mean to rag on Couch - I think it's a promising idea. But I ended up wasting several weeks going down this path only to throw it away because I didn't feel like it was production-ready.
marknadal 9y ago
If you are looking for something like CouchDB but only syncs partial subsets of the data you request (rather than the whole thing), try checking out https://github.com/amark/gun https://github.com/amark/gun (disclosure: I build it). However Postgres is probably the best database choice out there right now, so you made a good decision.

by tarr11 9y ago

sagelywizard 9y ago

The main feature which was added in 2.1 was the replication scheduler, which makes it possible to run millions of replications concurrently. That makes the database-per-user model much more feasible. As for throttling, the CouchDB replicator gracefully handles HTTP 429, but that wouldn't help with PouchDB-managed replications. I can't really speak on PouchDB, but maybe it's possible to limit the replication on the client side, since that's where it's managed?

tropshop 9y ago

> So, systems with many users (tens of thousands -> millions) would be stuck replicating an equal number of couch databases constantly I also ran into this when using `continuous: true` to create persistent replication, which suffers from this 1-to-1 resource problem. I now use a single listener on `_db_changes` and fire one-off replication to the aggregate db. I'm also keeping an eye on spiegel[1] I'm curious on how you handle offline support, sync conflicts, and multiple devices. I started with CouchDB 1.x and have been through enough growing pains to wonder how things would have been to stick with SQL, but I think the future is only brighter. CouchDB 2.1 looks great, and I hope the entire erlang ecosystem continues to see growth and adoption. [1] https://github.com/redgeoff/spiegel https://github.com/redgeoff/spiegel

tarr11 9y ago
I know that it is a common trope that Couch / Pouch seems to think that SQL databases don't work for offline replication and you need their "batteries-included" solution. However, it's not the only solution and every app in the app store that works offline has had to deal with it. Most of those are using SQL databases. In truth, what Couch does is force you to handle conflict events which gives you a convenient platform level place to put some of your automatic conflict/merge code. A lot of merge code is UI however and Couch does not help with that at all. Detecting conflicts is a subtle thing, but it's not necessarily a one-size-fits-all problem as couch would have us believe. I It also gives you versioning for free (which is pretty easy to implement in SQL) However, at the end of the day, every developer must make application level decisions on how to handle merge conflicts. Couch/Pouch does not obviate this - it just forces you to deal with it and lets you think about your application a little differently. Once you have to deal with merges and conflicts in your application code, it's more a matter of designing your application to have fewer conflicts (either through better merging, vector clocks, pessimistic locking, etc) I wish I could have used it but all my other data is in a battle-tested Postgres db, being nicely backed up regularly on Heroku without fail. My CouchDB was on a Google Cloud server, using a brand new replication framework as backup. I don't really mean to rag on Couch - I think it's a promising idea. But I ended up wasting several weeks going down this path only to throw it away because I didn't feel like it was production-ready.

marknadal 9y ago

If you are looking for something like CouchDB but only syncs partial subsets of the data you request (rather than the whole thing), try checking out https://github.com/amark/gun https://github.com/amark/gun (disclosure: I build it). However Postgres is probably the best database choice out there right now, so you made a good decision.