3 ms·

That's a really simple solution. It doesn't work for all kinds of data though. For some data you might want to have a more elaborate conflict resolution (e.g. m

by renke1 5y ago

That's a really simple solution. It doesn't work for all kinds of data though. For some data you might want to have a more elaborate conflict resolution (e.g. manual merge or using smart data structures like CRDT).

tehbeard 5y ago
I've briefly looked into CRDTs, but I have to ask, beyond a toy-implementation of a TODO list.... how much do they balloon the size of the data stored? Particularly for a complex document like a report with hundreds of fields and arbitrary sized lists for comments/observations?
- renke1 5y ago
  I think the most common CRDT libraries try hard to reduce the overhead of CRDTs. I am not expert but I could imagine that you could also remove some of the historical data if you know that all clients are reasonably up to date and if they aren't they have to discard their changes that are too old.
samwillis 5y ago
Earlier this year I experimented [1][2] with combining CRDTs using the incredible yJS[3] with PouchDB. It worked really well, completely magic syncing with full automatic handling of sync conflicts! (Although I have concerns about the sustainability of PouchDB, see my other comment[4]) 1: https://gist.github.com/samwillis/1465da23194d1ad480a5548458864077 https://gist.github.com/samwillis/1465da23194d1ad480a5548458... 2: https://discuss.yjs.dev/t/distributed-offline-editing-with-couch-pouchdb/340/7 https://discuss.yjs.dev/t/distributed-offline-editing-with-c... 3: https://yjs.dev https://yjs.dev 4: https://news.ycombinator.com/item?id=28690886 https://news.ycombinator.com/item?id=28690886
- renke1 5y ago
  I think my main problem with PouchDB and by extension CouchDB was that it seemed hard to add validation in the backend (including authentication/authorization). I remember having to build some kind of proxy that hooks into the CouchDB protocol to deny certain requests. I am pretty sure that's solved by now (or I was just asking the wrong questions back then).
  - patwolf 5y ago
    That was a problem I always had with replicating directly to CouchDB. They have added more authentication methods now, like proxy auth and JWT, so authorizing on a per-database basis isn't too bad. However, I gave up on CouchDB after my server kept getting hacked by crypto miners. I'm sure whatever exploit they were using has been patched, but I'm hesitant now to use a DB that's open to the world.
    - lytefm 5y ago
      If your CouchDB was open to the world, then that's definitely a configuration problem. Sure, earlier versions shipped with "admin party" enabled by default but the docs made it very clear to not do that in public.
  - lytefm 5y ago
    It's possible to use the same design docs both for client- and serverside validation. They don't look pretty, but maintaining them in readable JS and deploying them via CI works fine. Apart from Proxy Auth and JWT, just using basic Auth/session + a backend like Superlogin works for simple use cases. But sure, you'll want to set up rate limits etc using something like HaProxy once you have actual customer data on a CouchDB instance.

by renke1 5y ago

tehbeard 5y ago

I've briefly looked into CRDTs, but I have to ask, beyond a toy-implementation of a TODO list.... how much do they balloon the size of the data stored? Particularly for a complex document like a report with hundreds of fields and arbitrary sized lists for comments/observations?

renke1 5y ago
I think the most common CRDT libraries try hard to reduce the overhead of CRDTs. I am not expert but I could imagine that you could also remove some of the historical data if you know that all clients are reasonably up to date and if they aren't they have to discard their changes that are too old.

samwillis 5y ago

Earlier this year I experimented [1][2] with combining CRDTs using the incredible yJS[3] with PouchDB. It worked really well, completely magic syncing with full automatic handling of sync conflicts! (Although I have concerns about the sustainability of PouchDB, see my other comment[4]) 1: https://gist.github.com/samwillis/1465da23194d1ad480a5548458864077 https://gist.github.com/samwillis/1465da23194d1ad480a5548458... 2: https://discuss.yjs.dev/t/distributed-offline-editing-with-couch-pouchdb/340/7 https://discuss.yjs.dev/t/distributed-offline-editing-with-c... 3: https://yjs.dev https://yjs.dev 4: https://news.ycombinator.com/item?id=28690886 https://news.ycombinator.com/item?id=28690886

renke1 5y ago
I think my main problem with PouchDB and by extension CouchDB was that it seemed hard to add validation in the backend (including authentication/authorization). I remember having to build some kind of proxy that hooks into the CouchDB protocol to deny certain requests. I am pretty sure that's solved by now (or I was just asking the wrong questions back then).
- patwolf 5y ago
  That was a problem I always had with replicating directly to CouchDB. They have added more authentication methods now, like proxy auth and JWT, so authorizing on a per-database basis isn't too bad. However, I gave up on CouchDB after my server kept getting hacked by crypto miners. I'm sure whatever exploit they were using has been patched, but I'm hesitant now to use a DB that's open to the world.
  - lytefm 5y ago
    If your CouchDB was open to the world, then that's definitely a configuration problem. Sure, earlier versions shipped with "admin party" enabled by default but the docs made it very clear to not do that in public.
- lytefm 5y ago
  It's possible to use the same design docs both for client- and serverside validation. They don't look pretty, but maintaining them in readable JS and deploying them via CI works fine. Apart from Proxy Auth and JWT, just using basic Auth/session + a backend like Superlogin works for simple use cases. But sure, you'll want to set up rate limits etc using something like HaProxy once you have actual customer data on a CouchDB instance.