5 ms·

Hi, former Apache CouchDB guy, former Cloudant employee, and former Realm employee here… I (still) love CouchDB but at the end of the day I think its sync is a

Hi, former Apache CouchDB guy, former Cloudant employee, and former Realm employee here… I (still) love CouchDB but at the end of the day I think its sync is a better fit for server-to-server or web client-to-server use-cases, not mobile-to-server. There’s tons of stuff CouchDB sync doesn’t support that end up being a huge problem with mobile apps… Here are the top 3 things Realm adds in my opinion: great client-side performance, native models that are extremely easy to use, and conflict-free sync (not conflict resolution!). With my own eyes I’ve seen a generation of developers try CouchDB for mobile apps and then abandon it because of the limitation of its approach. Personally I’m glad to see someone try to bring more options to developers. I think this is the first general-case true sync (à la Google Docs) for mobile, and I hope it will continue to spur more iterations & innovation in other open-source projects as well.

avtar 10y ago
There’s tons of stuff CouchDB sync doesn’t support that end up being a huge problem with mobile apps… Here are the top 3 things Realm adds in my opinion: great client-side performance, native models that are extremely easy to use, and conflict-free sync (not conflict resolution!). Thanks for listing those potential benefits. Any idea if those CouchDB issues are covered in detail somewhere?
- daleharvey 10y ago
  I work on PouchDB, in terms of those points: "great client-side performance": PouchDB is more than fast enough for most use cases, its just a wrapper over indexedDB. (there are areas we can improve, particularly performance of map reduce / mango queries). "native models that are extremely easy to use": Not sure what that means, PouchDB and CouchDB use json, its native and easy to use in JavaScript "conflict-free sync (not conflict resolution!)": Certainly going to have to take a look at this, any conflict free sync protocol I have seen put a lot of limitations on how data within your application is structured but conflict resolution certainly is something that can be improved within Pouch / Couch. I do think there is a bunch we can (and are) improving in PouchDB and its great to see alternatives, will be looking into this further now. But I certainly do not agree that (C|P)ouchDB are not suited for mobile <-> web sync, thats the primary use case I work on Pouch for.
  - relics443 10y ago
    "native models that are extremely easy to use" This means native code (specifically not JSON). So in Java you would be dealing with POJOs, etc...
  - WA 10y ago
    A bit OT, but I have to compliment the person who wrote the docs[1] for PouchDB. They're quite entertaining and easy to understand. Well done! [1]: https://pouchdb.com/guides/ https://pouchdb.com/guides/
    - nolanl 10y ago
      Thanks, I'm that person. :) I tried to write it as a "missing manual" for Couch/Pouch, and had a lot of fun in the process! It needs some updates for newer stuff, though: pouchdb-find, custom builds, Chrome DevTools storage UI improvements, etc.
  - nolanl 10y ago
    (I also work on PouchDB.) For "conflict-free sync," according to the docs, this appears to be last-write-wins along with a special case for deletes: https://realm.io/docs/realm-object-server/#conflict-resolution https://realm.io/docs/realm-object-server/#conflict-resoluti... . (Incidentally Couch/Pouch also has a special case for deletes, but it does the opposite behavior by default – deletions lose to updates.) There's also some language in here about how "insertions in lists are ordered by time;" I'd be curious to know how they coordinate clock times across nodes. Ditto for "last update wins" – important to know how "last" is determined. In general, I tend to be skeptical of systems that hinge on last-write-wins, but LWW is definitely a model that works in a lot of cases where you're not too concerned about the possibility of losing data, and it's also a simpler mental model for the app developer.
    - astigsen 10y ago
      Realms approach to conflict resolution is very different from what you see in products like Couch. Realm is an _object_ database, so in contrast to document databases where conflict resolution happens very coarse grained at the document level, in Realm it happens at the object level (and actually all the way down into individual properties). This essentially means that the entire object graph is treated as one big CRDT [0], transparently handling conflict across individual objects, properties and relations. When talking about ordering by time, both in the context of LWW and insertions in lists, we are talking about vector clock time, not necessarily device time (or though that is taken into account as well). [0] https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type https://en.wikipedia.org/wiki/Conflict-free_replicated_data_...
ramses0 10y ago
Can you explain this statement: "There’s tons of stuff CouchDB sync doesn’t support that end up being a huge problem with mobile apps" You lay it out there and then talk about what Realm adds, but never what Couch lacks. And are you including PouchDB in your evaluation of CouchDB for "Client-Side Performance"? As I understand PouchDB is performant, uses JSON (pretty much the definition of "Native Models"), and allows server-arbitrated conflict resolution with stored revisions, making the simplest conflict resolution "last-in-wins" with user UI for "undo" (or any JS-object merge library). Thanks!
- andkon 10y ago
  I think you're misunderstanding what "Native Models" means in the context of iOS work. Saying that a database uses JSON immediately, to many ios devs, recalls years of writing very fragile json serialization code. Looking through CouchDB's iOS example apps confirms that yes, basically everything that isn't a string or int has to be serialized into JSON. That's a pretty enormous annoyance, and incorporates a lot of additional cognitive overhead when dealing with data that Realm simply handles properly for iOS from the beginning. I'm deeply relieved to see that they're launching something that does something about the fragile ios architectures built around JSON. It's similar to what they did with their mobile database: they actually listened to what mobile devs found annoying, and then they built something that completely sidestepped all the fragility problems that emerge when you wrap sqlite with an ORM. Instead, they made an approach that fit within our workflow, as opposed to taking us further from it.
  - ramses0 10y ago
    Ahhh, got it. When I hear "offline-first" my mind's been in webdev-mobile, not appstore-mobile. Thanks for clarifying!
  - janl 10y ago
    CouchDB doesn't have any iOS code, are you perhaps referring to Couchbase?
    - 10y ago
janl 10y ago
How is "conflict-free sync" done? The usual approach is last-write-wins, which is just a euphemism for "randomly losing user data", or CRDTs or Operational Transforms, which severely limit the data models one can sync. I'm happy to learn about a silver bullet here.
- striking 10y ago
  CRDT, but also vector clocks. https://news.ycombinator.com/item?id=12590753 https://news.ycombinator.com/item?id=12590753
  - gritzko 10y ago
    Some CRDTs employ vector clocks. CRDT is a veeeeery broad family of algorithms and data structures.
    - striking 10y ago
      Is that not what I stated (even if it was poorly phrased, and I can no longer edit it)?
- bigfish24 10y ago
  It is true that most Operational Transform systems have limited data model support, but that's an area Realm is quite unique on. The Realm Mobile Platform supports all of Realm database's current data types, such as native support for links and ordered lists. Also have plans for much more including: substring changes, counters and sets.
  - janl 10y ago
    Thanks, that's gonna be the limiting factor then, developers not finding a way to represent some of their storage needs on this platform. It's a fair trade-off, but hardly neither objectively worse or better than couch-sync. I'm excited to see where Realm goes. In my very unscientific approximation couch/git-style sync solves about 80% of general use cases, CRDT/OT cover another 10-15% and the rest are poor souls that have to do something custom. (All this excluding binary sync/rsync/BitTorrent/Dropbox.)
janl 10y ago
How is "conflict-free sync" done? The usual approach is last-write-wins, which is just a euphemism for "randomly losing user data", or CRDTs or Operational Transforms, which severely limit the data models one can sync. I'm happy to learn about a silver bullet here.
janl 10y ago
Oh, and while there is still room for improvement, we've improved the server-mobile sync case by an order of big-O-magnitude. And this is a first class design target for the future.

avtar 10y ago

There’s tons of stuff CouchDB sync doesn’t support that end up being a huge problem with mobile apps… Here are the top 3 things Realm adds in my opinion: great client-side performance, native models that are extremely easy to use, and conflict-free sync (not conflict resolution!). Thanks for listing those potential benefits. Any idea if those CouchDB issues are covered in detail somewhere?

daleharvey 10y ago
I work on PouchDB, in terms of those points: "great client-side performance": PouchDB is more than fast enough for most use cases, its just a wrapper over indexedDB. (there are areas we can improve, particularly performance of map reduce / mango queries). "native models that are extremely easy to use": Not sure what that means, PouchDB and CouchDB use json, its native and easy to use in JavaScript "conflict-free sync (not conflict resolution!)": Certainly going to have to take a look at this, any conflict free sync protocol I have seen put a lot of limitations on how data within your application is structured but conflict resolution certainly is something that can be improved within Pouch / Couch. I do think there is a bunch we can (and are) improving in PouchDB and its great to see alternatives, will be looking into this further now. But I certainly do not agree that (C|P)ouchDB are not suited for mobile <-> web sync, thats the primary use case I work on Pouch for.
- relics443 10y ago
  "native models that are extremely easy to use" This means native code (specifically not JSON). So in Java you would be dealing with POJOs, etc...
- WA 10y ago
  A bit OT, but I have to compliment the person who wrote the docs[1] for PouchDB. They're quite entertaining and easy to understand. Well done! [1]: https://pouchdb.com/guides/ https://pouchdb.com/guides/
  - nolanl 10y ago
    Thanks, I'm that person. :) I tried to write it as a "missing manual" for Couch/Pouch, and had a lot of fun in the process! It needs some updates for newer stuff, though: pouchdb-find, custom builds, Chrome DevTools storage UI improvements, etc.
- nolanl 10y ago
  (I also work on PouchDB.) For "conflict-free sync," according to the docs, this appears to be last-write-wins along with a special case for deletes: https://realm.io/docs/realm-object-server/#conflict-resolution https://realm.io/docs/realm-object-server/#conflict-resoluti... . (Incidentally Couch/Pouch also has a special case for deletes, but it does the opposite behavior by default – deletions lose to updates.) There's also some language in here about how "insertions in lists are ordered by time;" I'd be curious to know how they coordinate clock times across nodes. Ditto for "last update wins" – important to know how "last" is determined. In general, I tend to be skeptical of systems that hinge on last-write-wins, but LWW is definitely a model that works in a lot of cases where you're not too concerned about the possibility of losing data, and it's also a simpler mental model for the app developer.
  - astigsen 10y ago
    Realms approach to conflict resolution is very different from what you see in products like Couch. Realm is an _object_ database, so in contrast to document databases where conflict resolution happens very coarse grained at the document level, in Realm it happens at the object level (and actually all the way down into individual properties). This essentially means that the entire object graph is treated as one big CRDT [0], transparently handling conflict across individual objects, properties and relations. When talking about ordering by time, both in the context of LWW and insertions in lists, we are talking about vector clock time, not necessarily device time (or though that is taken into account as well). [0] https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type https://en.wikipedia.org/wiki/Conflict-free_replicated_data_...

ramses0 10y ago

Can you explain this statement: "There’s tons of stuff CouchDB sync doesn’t support that end up being a huge problem with mobile apps" You lay it out there and then talk about what Realm adds, but never what Couch lacks. And are you including PouchDB in your evaluation of CouchDB for "Client-Side Performance"? As I understand PouchDB is performant, uses JSON (pretty much the definition of "Native Models"), and allows server-arbitrated conflict resolution with stored revisions, making the simplest conflict resolution "last-in-wins" with user UI for "undo" (or any JS-object merge library). Thanks!

andkon 10y ago
I think you're misunderstanding what "Native Models" means in the context of iOS work. Saying that a database uses JSON immediately, to many ios devs, recalls years of writing very fragile json serialization code. Looking through CouchDB's iOS example apps confirms that yes, basically everything that isn't a string or int has to be serialized into JSON. That's a pretty enormous annoyance, and incorporates a lot of additional cognitive overhead when dealing with data that Realm simply handles properly for iOS from the beginning. I'm deeply relieved to see that they're launching something that does something about the fragile ios architectures built around JSON. It's similar to what they did with their mobile database: they actually listened to what mobile devs found annoying, and then they built something that completely sidestepped all the fragility problems that emerge when you wrap sqlite with an ORM. Instead, they made an approach that fit within our workflow, as opposed to taking us further from it.
- ramses0 10y ago
  Ahhh, got it. When I hear "offline-first" my mind's been in webdev-mobile, not appstore-mobile. Thanks for clarifying!
- janl 10y ago
  CouchDB doesn't have any iOS code, are you perhaps referring to Couchbase?
  - 10y ago

janl 10y ago

How is "conflict-free sync" done? The usual approach is last-write-wins, which is just a euphemism for "randomly losing user data", or CRDTs or Operational Transforms, which severely limit the data models one can sync. I'm happy to learn about a silver bullet here.

striking 10y ago
CRDT, but also vector clocks. https://news.ycombinator.com/item?id=12590753 https://news.ycombinator.com/item?id=12590753
- gritzko 10y ago
  Some CRDTs employ vector clocks. CRDT is a veeeeery broad family of algorithms and data structures.
  - striking 10y ago
    Is that not what I stated (even if it was poorly phrased, and I can no longer edit it)?
bigfish24 10y ago
It is true that most Operational Transform systems have limited data model support, but that's an area Realm is quite unique on. The Realm Mobile Platform supports all of Realm database's current data types, such as native support for links and ordered lists. Also have plans for much more including: substring changes, counters and sets.
- janl 10y ago
  Thanks, that's gonna be the limiting factor then, developers not finding a way to represent some of their storage needs on this platform. It's a fair trade-off, but hardly neither objectively worse or better than couch-sync. I'm excited to see where Realm goes. In my very unscientific approximation couch/git-style sync solves about 80% of general use cases, CRDT/OT cover another 10-15% and the rest are poor souls that have to do something custom. (All this excluding binary sync/rsync/BitTorrent/Dropbox.)

janl 10y ago

Oh, and while there is still room for improvement, we've improved the server-mobile sync case by an order of big-O-magnitude. And this is a first class design target for the future.