3 ms·

I don't understand this obsession with offline support for collaborative realtime apps. There is an idea going around that you can magically sync everything usi

I don't understand this obsession with offline support for collaborative realtime apps. There is an idea going around that you can magically sync everything using the same algorithm. Accurate offline collaboration is impossible without some form of user-assisted conflict resolution - The core problem is that automatic sync algorithms cannot capture the collaborative intent of users. For example, assume that you have a list of items and two users are interacting with this list while offline. Let's say that one user called 'John' coincidentally deletes all items which start with the letter 'A' but, while also offline, the other user 'Steve' renames one of those 'A' items such that it now starts with the letter 'B'. There is a problem here, because if John comes back online first; all items starting with 'A' will get deleted. So when Bob comes online, either his rename operation will fail because the system cannot rename a non-existant item or it will re-create the item which John explicitly deleted (but this time with a new name - Which goes against John's intention). The system can never know for sure the intent of both users so it cannot automatically resolve conflicts (at least not accurately). In some cases, the benefits outweigh the disadvantages but it's important to be aware of those disadvantages and that there are no one-size-fits-all solutions which work perfectly with all kinds of data.

ZitchDog 11y ago
The authors of CouchDB understand this, too. The document would be marked with a conflict, and the application could try to resolve this conflict if it makes sense, or surface a UI for the user to resolve the changes.
wereHamster 11y ago
Or you design the protocol (or rather, the operations which are applied to the data) so that conflicts can be trivially resolved. In your example you could have a single operation, SET, which sets a field at a particular address to the given value. User John sends a list of SET operations, all in the form of 'SET "<item X>.deleted" true' for all items 'X' which he wants to get rid of. Steve sends a list of SET operations in the form of 'SET "<item X>.name" ???', which don't conflict with what John sent. The key point is to have stable addresses (eg. never delete objects so that addresses within it remain valid forever). And a small set of fundamental operations on them where conflicts can automatically be resolved by the computer (you get get by for a long time with just SET, and if you have just SET then conflict resolution is trivial). Note that in that system there is no notion of a 'Delete all users whose names begin with 'A'" operation. That's a very complex operation with very narrow use case. And it requires a lot of logic to handle all the potential conflicts with other such complex operations.
- iampims 11y ago
  That's exactly what CRDTs are all about. The garbage collection problem is a real issue in practice.
mbrock 11y ago
"I don't understand this obsession with offline support for collaborative realtime apps." You're already offline. Conflicts can always happen. Your 3G disconnects frequently. If you want a robust app, and especially if your target audience is often on the road, why wouldn't you be "obsessed" with offline sync for collaboration? And if you really need user-guided conflict resolution, you can just make that.
eloff 11y ago
Yeah, this is right on the money. I've worked in data sync the last four years, supporting an app that's used in rural parts of the developing world. Unfortunately offline sync was the only available option. What I learned from the experience is that sync is a super hard problem, that conflicts are inevitable, and that you should sync as much information and context as possible. Sync the higher level business actions, not low level data. Because when there's a conflict, you need the extra context to allow the user to resolve it in a sensible way. If you only synced low level data, you frequently find it's impossible to recover from some kind of conflicts.
- pgte 11y ago
  PouchDB has great support for getting conflicts and merging them, it doesn't have to be dealt with at the sync layer. Since the client is the owner of the PouchDB database, it has direct access to conflicts and can fix them on the spot. Your data should be devised to be easily mergeable. Some document schemas are more easibly mergeable than others, CRDTs being the easiest extreme I know.

ZitchDog 11y ago

The authors of CouchDB understand this, too. The document would be marked with a conflict, and the application could try to resolve this conflict if it makes sense, or surface a UI for the user to resolve the changes.

wereHamster 11y ago

Or you design the protocol (or rather, the operations which are applied to the data) so that conflicts can be trivially resolved. In your example you could have a single operation, SET, which sets a field at a particular address to the given value. User John sends a list of SET operations, all in the form of 'SET "<item X>.deleted" true' for all items 'X' which he wants to get rid of. Steve sends a list of SET operations in the form of 'SET "<item X>.name" ???', which don't conflict with what John sent. The key point is to have stable addresses (eg. never delete objects so that addresses within it remain valid forever). And a small set of fundamental operations on them where conflicts can automatically be resolved by the computer (you get get by for a long time with just SET, and if you have just SET then conflict resolution is trivial). Note that in that system there is no notion of a 'Delete all users whose names begin with 'A'" operation. That's a very complex operation with very narrow use case. And it requires a lot of logic to handle all the potential conflicts with other such complex operations.

iampims 11y ago
That's exactly what CRDTs are all about. The garbage collection problem is a real issue in practice.

mbrock 11y ago

"I don't understand this obsession with offline support for collaborative realtime apps." You're already offline. Conflicts can always happen. Your 3G disconnects frequently. If you want a robust app, and especially if your target audience is often on the road, why wouldn't you be "obsessed" with offline sync for collaboration? And if you really need user-guided conflict resolution, you can just make that.

eloff 11y ago

Yeah, this is right on the money. I've worked in data sync the last four years, supporting an app that's used in rural parts of the developing world. Unfortunately offline sync was the only available option. What I learned from the experience is that sync is a super hard problem, that conflicts are inevitable, and that you should sync as much information and context as possible. Sync the higher level business actions, not low level data. Because when there's a conflict, you need the extra context to allow the user to resolve it in a sensible way. If you only synced low level data, you frequently find it's impossible to recover from some kind of conflicts.

pgte 11y ago
PouchDB has great support for getting conflicts and merging them, it doesn't have to be dealt with at the sync layer. Since the client is the owner of the PouchDB database, it has direct access to conflicts and can fix them on the spot. Your data should be devised to be easily mergeable. Some document schemas are more easibly mergeable than others, CRDTs being the easiest extreme I know.