8 ms·

Building realtime collaborative offline-first apps with React, Redux, PouchDB

67 points by tilt 11y ago

mizzao 11y ago
Seems like Meteor can slot in another database next to the current MongoDB+MiniMongo implementation?
OldSchoolJohnny 11y ago
"But yes, the server has no control over what data is being written. This is why this type of architecture only serves for personal / shared databases, not for public records." A solution that doesn't take into effect security from the start is pretty limited for any real world usage beyond a simple personal data app in which case why over-architect it in the first place?
- daleharvey 11y ago
  That generalization isnt totally accurate, CouchDB has in built ability to validate data that gets written to it, and PouchDB was built specifically to make that type of control easy and powerful.
- scoot 11y ago
  Yup. I put together a successful proof-of-concept with React and PouchDB / CouchDB, but quickly concluded the lack of granular security was its Achilles heel. I switched to Meteor with GroundDB for offline support, and haven't looked back.
- pgte 11y ago
  This can be addressed in CouchDB, by creating a validating function that validates all the incoming updates. Besides that, I'm addressing this via the architecture (there's an aside in the article where I cover this briefly), which will be a subject of a future article: Central records shouldn't be directly changed by a user. That's now how central record systems should work. Instead, we should view any document as a request to get or change the central records, to be processed by a clerk. The clerk then changes the state of the document according to the permissions and results.
rfrey 11y ago
A couple of comments since I've spent the last month building out this exact architecture. First, the author doesn't really address security and authorization, in fact they gloss over it. The typical solution with CouchDB/PouchDB seems to be per-user databases. That sounds horrendous to a SQL guy (me), but it turns out that CouchDB can happily handle hundreds of thousands of databases. In my current application I have found couchperuser very useful. It's an Erlang Couchdb plugin that creates a database with appropriate permissions every time you create an entry in the _users database. (Access to the _users database is restricted to an admin account so is done via a parallel web service.) If users are accessing CouchDB directly, of course you gotta worry about what horrors they're stuffing into your lovely pristine database. Control over that is provided via Javascript validation functions that are run on every document save. Those are saved into the per-user database at account creation time. The biggest downside to this architecture that I've found is that there is two sources of truth on the client - the Redux store and the PouchDB database.
- pgte 11y ago
  For the client there's only one source of truth: the Redux store. The PouchDB store is a side effect of changes to the Redux store, no need for the client to worry about these.
jondubois 11y ago
I don't understand this obsession with offline support for collaborative realtime apps. There is an idea going around that you can magically sync everything using the same algorithm. Accurate offline collaboration is impossible without some form of user-assisted conflict resolution - The core problem is that automatic sync algorithms cannot capture the collaborative intent of users. For example, assume that you have a list of items and two users are interacting with this list while offline. Let's say that one user called 'John' coincidentally deletes all items which start with the letter 'A' but, while also offline, the other user 'Steve' renames one of those 'A' items such that it now starts with the letter 'B'. There is a problem here, because if John comes back online first; all items starting with 'A' will get deleted. So when Bob comes online, either his rename operation will fail because the system cannot rename a non-existant item or it will re-create the item which John explicitly deleted (but this time with a new name - Which goes against John's intention). The system can never know for sure the intent of both users so it cannot automatically resolve conflicts (at least not accurately). In some cases, the benefits outweigh the disadvantages but it's important to be aware of those disadvantages and that there are no one-size-fits-all solutions which work perfectly with all kinds of data.
- ZitchDog 11y ago
  The authors of CouchDB understand this, too. The document would be marked with a conflict, and the application could try to resolve this conflict if it makes sense, or surface a UI for the user to resolve the changes.
- wereHamster 11y ago
  Or you design the protocol (or rather, the operations which are applied to the data) so that conflicts can be trivially resolved. In your example you could have a single operation, SET, which sets a field at a particular address to the given value. User John sends a list of SET operations, all in the form of 'SET "<item X>.deleted" true' for all items 'X' which he wants to get rid of. Steve sends a list of SET operations in the form of 'SET "<item X>.name" ???', which don't conflict with what John sent. The key point is to have stable addresses (eg. never delete objects so that addresses within it remain valid forever). And a small set of fundamental operations on them where conflicts can automatically be resolved by the computer (you get get by for a long time with just SET, and if you have just SET then conflict resolution is trivial). Note that in that system there is no notion of a 'Delete all users whose names begin with 'A'" operation. That's a very complex operation with very narrow use case. And it requires a lot of logic to handle all the potential conflicts with other such complex operations.

67 points by tilt 11y ago

Seems like Meteor can slot in another database next to the current MongoDB+MiniMongo implementation?

"But yes, the server has no control over what data is being written. This is why this type of architecture only serves for personal / shared databases, not for public records." A solution that doesn't take into effect security from the start is pretty limited for any real world usage beyond a simple personal data app in which case why over-architect it in the first place?

daleharvey 11y ago
That generalization isnt totally accurate, CouchDB has in built ability to validate data that gets written to it, and PouchDB was built specifically to make that type of control easy and powerful.
scoot 11y ago
Yup. I put together a successful proof-of-concept with React and PouchDB / CouchDB, but quickly concluded the lack of granular security was its Achilles heel. I switched to Meteor with GroundDB for offline support, and haven't looked back.
pgte 11y ago
This can be addressed in CouchDB, by creating a validating function that validates all the incoming updates. Besides that, I'm addressing this via the architecture (there's an aside in the article where I cover this briefly), which will be a subject of a future article: Central records shouldn't be directly changed by a user. That's now how central record systems should work. Instead, we should view any document as a request to get or change the central records, to be processed by a clerk. The clerk then changes the state of the document according to the permissions and results.

rfrey 11y ago

A couple of comments since I've spent the last month building out this exact architecture. First, the author doesn't really address security and authorization, in fact they gloss over it. The typical solution with CouchDB/PouchDB seems to be per-user databases. That sounds horrendous to a SQL guy (me), but it turns out that CouchDB can happily handle hundreds of thousands of databases. In my current application I have found couchperuser very useful. It's an Erlang Couchdb plugin that creates a database with appropriate permissions every time you create an entry in the _users database. (Access to the _users database is restricted to an admin account so is done via a parallel web service.) If users are accessing CouchDB directly, of course you gotta worry about what horrors they're stuffing into your lovely pristine database. Control over that is provided via Javascript validation functions that are run on every document save. Those are saved into the per-user database at account creation time. The biggest downside to this architecture that I've found is that there is two sources of truth on the client - the Redux store and the PouchDB database.

pgte 11y ago
For the client there's only one source of truth: the Redux store. The PouchDB store is a side effect of changes to the Redux store, no need for the client to worry about these.

jondubois 11y ago

I don't understand this obsession with offline support for collaborative realtime apps. There is an idea going around that you can magically sync everything using the same algorithm. Accurate offline collaboration is impossible without some form of user-assisted conflict resolution - The core problem is that automatic sync algorithms cannot capture the collaborative intent of users. For example, assume that you have a list of items and two users are interacting with this list while offline. Let's say that one user called 'John' coincidentally deletes all items which start with the letter 'A' but, while also offline, the other user 'Steve' renames one of those 'A' items such that it now starts with the letter 'B'. There is a problem here, because if John comes back online first; all items starting with 'A' will get deleted. So when Bob comes online, either his rename operation will fail because the system cannot rename a non-existant item or it will re-create the item which John explicitly deleted (but this time with a new name - Which goes against John's intention). The system can never know for sure the intent of both users so it cannot automatically resolve conflicts (at least not accurately). In some cases, the benefits outweigh the disadvantages but it's important to be aware of those disadvantages and that there are no one-size-fits-all solutions which work perfectly with all kinds of data.

ZitchDog 11y ago
The authors of CouchDB understand this, too. The document would be marked with a conflict, and the application could try to resolve this conflict if it makes sense, or surface a UI for the user to resolve the changes.
wereHamster 11y ago
Or you design the protocol (or rather, the operations which are applied to the data) so that conflicts can be trivially resolved. In your example you could have a single operation, SET, which sets a field at a particular address to the given value. User John sends a list of SET operations, all in the form of 'SET "<item X>.deleted" true' for all items 'X' which he wants to get rid of. Steve sends a list of SET operations in the form of 'SET "<item X>.name" ???', which don't conflict with what John sent. The key point is to have stable addresses (eg. never delete objects so that addresses within it remain valid forever). And a small set of fundamental operations on them where conflicts can automatically be resolved by the computer (you get get by for a long time with just SET, and if you have just SET then conflict resolution is trivial). Note that in that system there is no notion of a 'Delete all users whose names begin with 'A'" operation. That's a very complex operation with very narrow use case. And it requires a lot of logic to handle all the potential conflicts with other such complex operations.