4 ms·

You should also check out Yjs/Yrs or Automerge which implement merging data structures using CRDTs. Most people think of them as useful for colabrotavive real-t

You should also check out Yjs/Yrs or Automerge which implement merging data structures using CRDTs. Most people think of them as useful for colabrotavive real-time editing from multiple users but they are just as good at offline merges. They are perfect for situation like you describe where you have one user with multiple devices. Apple Notes use’s CRDTs internally for exactly this. Last year I experimented with an app architecture that used CouchDB/PouchDB for for synchronising data for a single user, multi device app. Then using Yjs to merge the conflicting edits - it worked incredible well. If I had the time I would love to build a Yjs/CRDT native CouchDB like database that could use the Yjs state vectors as a wire protocol for syncing…

dasz 5y ago
If there's some more depth or otherwise more detail then you should consider writing an article or blog post. Sounds interesting.
- samwillis 5y ago
  If have a little free time someday I'm going to spend a day polishing what I was building and Open-source it. It was a kind of investigation into a few different ideas, it was the synced data store which was the more interesting thing that came out of it. This is the very rough code behind the PouchDB/Yjs datastore. Effectively each Pouch/Couch document is actually "managed" by Yjs, all changes/operations via it. It then saves the binary Yjs blob as an attachment on the Pouch document with the current Yjs state exported as JSON for the main Pouch document. This gives you all the indexing/search you get with Pouch/Couch but with automatic merging of conflicting edits. https://gist.github.com/samwillis/1465da23194d1ad480a5548458864077 https://gist.github.com/samwillis/1465da23194d1ad480a5548458... and a tread about it: https://discuss.yjs.dev/t/distributed-offline-editing-with-couch-pouchdb/340/2 https://discuss.yjs.dev/t/distributed-offline-editing-with-c... Ultimately though I don't think PouchDB is a good platform for this, building something that is native Yjs would be much better. If anyone is interested I would love to hear from them though!
  - WorldMaker 4y ago
    I'm also interested in following updates to your approach here. Something that stands out immediately to me is that reliance on binary attachments. In my own CouchDB ecosystem work binary attachments have turned out to be just about the worst part of the ecosystem. PouchDB stores them pretty reliably, but every other CouchDB ecosystem database (Couchbase, Cloudant) including different versions of CouchDB itself (1.x is different from 2.x is different from 3.x in all sorts of ways) all have very different behavior when synchronizing attachments, the allowed size of attachments, the allowed types of attachments, the allowed characters in attachment names, and in general the sync protocol itself is prone to failures/timeouts with large attachments that are tough to work around because the break in the middle of replications. The number of times I've had to delete an attachment that PouchDB stored just fine to get a sync operation to complete with another server has been way too many already. I've had to build bespoke attachment sync tools because I haven't been able to rely on attachments working in the CouchDB ecosystem. Probably more evidence to back up your gut instinct that a native Yjs-oriented store may be better in the long term. (Admittedly, too, I've been thinking that I need to replace the CouchDB ecosystem as a whole. PouchDB is great, but the flux I've seen in the Apache CouchDB project and the issues I've had with the managed service providers especially Cloudant after IBM makes it really hard to recommend the ecosystem. Overall it seems unhealthy/in-decline, which is sad when the core sync infrastructure seems so nice to work with when it works.)

dasz 5y ago

If there's some more depth or otherwise more detail then you should consider writing an article or blog post. Sounds interesting.

samwillis 5y ago
If have a little free time someday I'm going to spend a day polishing what I was building and Open-source it. It was a kind of investigation into a few different ideas, it was the synced data store which was the more interesting thing that came out of it. This is the very rough code behind the PouchDB/Yjs datastore. Effectively each Pouch/Couch document is actually "managed" by Yjs, all changes/operations via it. It then saves the binary Yjs blob as an attachment on the Pouch document with the current Yjs state exported as JSON for the main Pouch document. This gives you all the indexing/search you get with Pouch/Couch but with automatic merging of conflicting edits. https://gist.github.com/samwillis/1465da23194d1ad480a5548458864077 https://gist.github.com/samwillis/1465da23194d1ad480a5548458... and a tread about it: https://discuss.yjs.dev/t/distributed-offline-editing-with-couch-pouchdb/340/2 https://discuss.yjs.dev/t/distributed-offline-editing-with-c... Ultimately though I don't think PouchDB is a good platform for this, building something that is native Yjs would be much better. If anyone is interested I would love to hear from them though!
- WorldMaker 4y ago
  I'm also interested in following updates to your approach here. Something that stands out immediately to me is that reliance on binary attachments. In my own CouchDB ecosystem work binary attachments have turned out to be just about the worst part of the ecosystem. PouchDB stores them pretty reliably, but every other CouchDB ecosystem database (Couchbase, Cloudant) including different versions of CouchDB itself (1.x is different from 2.x is different from 3.x in all sorts of ways) all have very different behavior when synchronizing attachments, the allowed size of attachments, the allowed types of attachments, the allowed characters in attachment names, and in general the sync protocol itself is prone to failures/timeouts with large attachments that are tough to work around because the break in the middle of replications. The number of times I've had to delete an attachment that PouchDB stored just fine to get a sync operation to complete with another server has been way too many already. I've had to build bespoke attachment sync tools because I haven't been able to rely on attachments working in the CouchDB ecosystem. Probably more evidence to back up your gut instinct that a native Yjs-oriented store may be better in the long term. (Admittedly, too, I've been thinking that I need to replace the CouchDB ecosystem as a whole. PouchDB is great, but the flux I've seen in the Apache CouchDB project and the issues I've had with the managed service providers especially Cloudant after IBM makes it really hard to recommend the ecosystem. Overall it seems unhealthy/in-decline, which is sad when the core sync infrastructure seems so nice to work with when it works.)