4 ms·

I actually have a need for something like this. I wrote a single-user flashcard app which supports multi-device sync. The way I sync is having each device write

by allenu 4y ago

I actually have a need for something like this. I wrote a single-user flashcard app which supports multi-device sync. The way I sync is having each device write to its own journal file a list of commands, such as "create card" or "update card with ID 1234" or "insert card evaluation". To "sync", each device just pushes up its journal to a common file store and pulls down each other device's journals. Then it just plays back all the edits in timestamp order. (It works fine for single-user since you're very unlikely to be using two devices at the exact same time.) Anyway, I never did end up implementing an optimal "update card with ID 1234". If you edit a flashcard's front text, instead of updating the front text field alone, I ended up just writing out an entirely new copy of the flashcard in the journal (with the ID 1234 encoded). Super inefficient. I was toying with the idea of coming up with a simpler format that does basically what this JSON patch does, where you an just encode "update object 1234's field foo.bar.baz to string ABC". I figured I'd only need a handful of operations, to insert a new object, replace a node in an object's property tree given a key path, delete an object, append to an array property, etc. If anyone has other solutions other than JSON Patch, I'd love to hear about them.

rendall 4y ago
JSON Patch seems like a good solution.
samwillis 4y ago
You should also check out Yjs/Yrs or Automerge which implement merging data structures using CRDTs. Most people think of them as useful for colabrotavive real-time editing from multiple users but they are just as good at offline merges. They are perfect for situation like you describe where you have one user with multiple devices. Apple Notes use’s CRDTs internally for exactly this. Last year I experimented with an app architecture that used CouchDB/PouchDB for for synchronising data for a single user, multi device app. Then using Yjs to merge the conflicting edits - it worked incredible well. If I had the time I would love to build a Yjs/CRDT native CouchDB like database that could use the Yjs state vectors as a wire protocol for syncing…
- dasz 4y ago
  If there's some more depth or otherwise more detail then you should consider writing an article or blog post. Sounds interesting.
  - samwillis 4y ago
    If have a little free time someday I'm going to spend a day polishing what I was building and Open-source it. It was a kind of investigation into a few different ideas, it was the synced data store which was the more interesting thing that came out of it. This is the very rough code behind the PouchDB/Yjs datastore. Effectively each Pouch/Couch document is actually "managed" by Yjs, all changes/operations via it. It then saves the binary Yjs blob as an attachment on the Pouch document with the current Yjs state exported as JSON for the main Pouch document. This gives you all the indexing/search you get with Pouch/Couch but with automatic merging of conflicting edits. https://gist.github.com/samwillis/1465da23194d1ad480a5548458864077 https://gist.github.com/samwillis/1465da23194d1ad480a5548458... and a tread about it: https://discuss.yjs.dev/t/distributed-offline-editing-with-couch-pouchdb/340/2 https://discuss.yjs.dev/t/distributed-offline-editing-with-c... Ultimately though I don't think PouchDB is a good platform for this, building something that is native Yjs would be much better. If anyone is interested I would love to hear from them though!
    - WorldMaker 4y ago
      I'm also interested in following updates to your approach here. Something that stands out immediately to me is that reliance on binary attachments. In my own CouchDB ecosystem work binary attachments have turned out to be just about the worst part of the ecosystem. PouchDB stores them pretty reliably, but every other CouchDB ecosystem database (Couchbase, Cloudant) including different versions of CouchDB itself (1.x is different from 2.x is different from 3.x in all sorts of ways) all have very different behavior when synchronizing attachments, the allowed size of attachments, the allowed types of attachments, the allowed characters in attachment names, and in general the sync protocol itself is prone to failures/timeouts with large attachments that are tough to work around because the break in the middle of replications. The number of times I've had to delete an attachment that PouchDB stored just fine to get a sync operation to complete with another server has been way too many already. I've had to build bespoke attachment sync tools because I haven't been able to rely on attachments working in the CouchDB ecosystem. Probably more evidence to back up your gut instinct that a native Yjs-oriented store may be better in the long term. (Admittedly, too, I've been thinking that I need to replace the CouchDB ecosystem as a whole. PouchDB is great, but the flux I've seen in the Apache CouchDB project and the issues I've had with the managed service providers especially Cloudant after IBM makes it really hard to recommend the ecosystem. Overall it seems unhealthy/in-decline, which is sad when the core sync infrastructure seems so nice to work with when it works.)
LukeEF 4y ago
There is a diff and patch API described here - the underlying database (TerminusDB) also has the concept of merging records, which could fit to your use case: https://terminusdb.com/docs/index/json-diff-and-patch https://terminusdb.com/docs/index/json-diff-and-patch

by allenu 4y ago

rendall 4y ago

JSON Patch seems like a good solution.

samwillis 4y ago

You should also check out Yjs/Yrs or Automerge which implement merging data structures using CRDTs. Most people think of them as useful for colabrotavive real-time editing from multiple users but they are just as good at offline merges. They are perfect for situation like you describe where you have one user with multiple devices. Apple Notes use’s CRDTs internally for exactly this. Last year I experimented with an app architecture that used CouchDB/PouchDB for for synchronising data for a single user, multi device app. Then using Yjs to merge the conflicting edits - it worked incredible well. If I had the time I would love to build a Yjs/CRDT native CouchDB like database that could use the Yjs state vectors as a wire protocol for syncing…

dasz 4y ago
If there's some more depth or otherwise more detail then you should consider writing an article or blog post. Sounds interesting.
- samwillis 4y ago
  If have a little free time someday I'm going to spend a day polishing what I was building and Open-source it. It was a kind of investigation into a few different ideas, it was the synced data store which was the more interesting thing that came out of it. This is the very rough code behind the PouchDB/Yjs datastore. Effectively each Pouch/Couch document is actually "managed" by Yjs, all changes/operations via it. It then saves the binary Yjs blob as an attachment on the Pouch document with the current Yjs state exported as JSON for the main Pouch document. This gives you all the indexing/search you get with Pouch/Couch but with automatic merging of conflicting edits. https://gist.github.com/samwillis/1465da23194d1ad480a5548458864077 https://gist.github.com/samwillis/1465da23194d1ad480a5548458... and a tread about it: https://discuss.yjs.dev/t/distributed-offline-editing-with-couch-pouchdb/340/2 https://discuss.yjs.dev/t/distributed-offline-editing-with-c... Ultimately though I don't think PouchDB is a good platform for this, building something that is native Yjs would be much better. If anyone is interested I would love to hear from them though!
  - WorldMaker 4y ago
    I'm also interested in following updates to your approach here. Something that stands out immediately to me is that reliance on binary attachments. In my own CouchDB ecosystem work binary attachments have turned out to be just about the worst part of the ecosystem. PouchDB stores them pretty reliably, but every other CouchDB ecosystem database (Couchbase, Cloudant) including different versions of CouchDB itself (1.x is different from 2.x is different from 3.x in all sorts of ways) all have very different behavior when synchronizing attachments, the allowed size of attachments, the allowed types of attachments, the allowed characters in attachment names, and in general the sync protocol itself is prone to failures/timeouts with large attachments that are tough to work around because the break in the middle of replications. The number of times I've had to delete an attachment that PouchDB stored just fine to get a sync operation to complete with another server has been way too many already. I've had to build bespoke attachment sync tools because I haven't been able to rely on attachments working in the CouchDB ecosystem. Probably more evidence to back up your gut instinct that a native Yjs-oriented store may be better in the long term. (Admittedly, too, I've been thinking that I need to replace the CouchDB ecosystem as a whole. PouchDB is great, but the flux I've seen in the Apache CouchDB project and the issues I've had with the managed service providers especially Cloudant after IBM makes it really hard to recommend the ecosystem. Overall it seems unhealthy/in-decline, which is sad when the core sync infrastructure seems so nice to work with when it works.)

LukeEF 4y ago

There is a diff and patch API described here - the underlying database (TerminusDB) also has the concept of merging records, which could fit to your use case: https://terminusdb.com/docs/index/json-diff-and-patch https://terminusdb.com/docs/index/json-diff-and-patch