6 ms·

Its a good article but it should be noted that this very much is a web-centric view of Offline First and its challenges. When I say web-centric, I mean as oppos

Its a good article but it should be noted that this very much is a web-centric view of Offline First and its challenges. When I say web-centric, I mean as opposed to offline-first on a mobile app. Native apps dont deal with the same issues around storage, and are actually _much_ more performant overall. If you haven't done an offline-first app, I highly recommend it. The experience is magical. You can fly around your app at the speed of the users touch. Content is magically loaded as soon as its clicked on. Its amazing as a user. As for similarities, conflict resolution is a universal problem and is either the most difficult or second most difficult problem [1] to solve. What makes this more difficult is that there is no one-size-fits-all for this. You need to have a deep, nuanced understanding of your system and what makes sense for your use case in terms of resolution strategies. Then you need to implement them, which is not easy especially if your backend is bog-standard REST on a classic SQL datastore. I've enjoyed reading the past 2 rxdb articles on this (the one mentioned here as well as the one from a couple days ago [2]). Its great to have more content on this publicly available, when I was getting into offline-first I only had a couple options. [1] - At the end of the day, offline-first is a whole bunch of caching, and so you end up needing to deep dive into cache invalidation strategies which we all know is a hairy problem. [2] - https://news.ycombinator.com/item?id=28690427 https://news.ycombinator.com/item?id=28690427

Trufa 5y ago
Yeah,I'm trying to implement an electron offline first app that syncs, there seems to readymade solution. Stuff like https://github.com/aerogear/offix https://github.com/aerogear/offix seem to be in the right direction of what I'm looking for but not nearly mature enough. I don't want to pu to much effort on the app so I would like something more or less ready made, preferably with graphql apis. Any suggestions welcome.
- JamesSwift 5y ago
  I would hesitate to use any client-server setup that isnt also your primary data store. So, if you have an "offline aware, multi-tenant" store like CouchDB but you still need to sync to the primary store which is SQL, then you lose a lot of context and awareness around conflict resolution. If you are going to eventually sync to the other store, I would say only use Couch on the frontend, and do the syncing/resolution from the perspective of the client, since the client knows what it was trying to do and how best to notify on conflict. The "better" option is to have an offline-aware client-server component (e.g. Realm) which is the primary store as well. This eliminates the sync and so all conflict resolution stays in the same system with well defined semantics.
  - Trufa 5y ago
    Thanks for the recommendation, I'm a little bit skeptical of MongoDB, but realm looks nice, Firebase I'm a little reluctant on being so dependant on google, but otherwise looks good.
    - BackBlast 5y ago
      PouchDB works well as the primary client store and can sync to CouchDB in the cloud. IMHO this is one of the more mature combinations that gives you a ready client side document database that takes care of data replication for you.
      - WorldMaker 5y ago
        It's just unfortunate that running CouchDB in the cloud seems increasingly perilous. Since IBM acquired Cloudant, it's much tougher to run CouchDB "at scale PaaS" on any data center other than IBM's (for obvious reasons), whereas early Cloudant had robust Azure and AWS support for years. I wish Couchbase were more helpful practically than they try to present themselves theoretically. Even if their products weren't so expensive the impedance mismatch between their version of CouchDB's sync APIs and their own APIs seems to increase by the year, and is pretty noticeable in how different it works from a PouchDB standpoint and how easy it is to break sync. (Impedance mismatches in allowable database names and _id keys are huge on their own that have massive repercussions in application design.) Even CouchDB is not CouchDB anymore with impedance mismatches of its own between versions. On the one hand it's good that Cloudant upstreamed a lot of their cluster management tools directly into Apache CouchDB 2+ (even as they made their PaaS offering IBM Cloud only [or whatever it's name of the month is]), but huge architectural changes below the covers in CouchDB 3+ start to present their own sync issues akin to but distinct from Couchbase's (and even some of Cloudant's as they seem [?] to be diverging again into their own 2+ fork after all that work upstreaming stuff?). More than ever, Azure CosmosDB's focus on bare minimum MongoDB capability and not supporting anything like CouchDB sync, despite having close the same raw ingredients (Cosmos' change feed looks a lot more like Couch's, but is just missing a couple subtle things to make it directly and immediately useful for Couch replication) seems like a "CouchDB is dead and not worth supporting" signal from Microsoft. Unfortunately, I think PouchDB <=> CouchDB replication has past the "mature" point to the "decrepit" and "falling apart" stage, maybe to the point of "evolutionary dead end" if I'm feeling strongly pessimistic enough, and I've been for years trying to figure what to replace it with.
- firebase-user 5y ago
  You should try using Firebase. It handles the data-sync for you, and it applies all updates locally first so that your app feels snappy.
  - danuker 5y ago
    Ah yes. Convenience is how Big Tech gets all data to pass through it.
  - JamesSwift 5y ago
    I mentioned Realm because thats what Im familiar with but yeah Firebase is also a good choice if you want a client-server setup.
  - code-is-code 5y ago
    Firebase works until you want a different conflict resolution then last-write-wins. It is also only offline first if the user is authenticated. Otherwise you need a connection to the servers before using the local state.
  - jamil7 5y ago
    Firebase is offline-resistant more so than offline-first I’d argue.
  - LAC-Tech 5y ago
    Last write wins is not handling data sync, it's washing your hands of it. Your users will be frustrated.
  - davidzweig 5y ago
    The (JS) frontend SDK can be set to either online or offline mode. When in online mode, it seems to insist on downloading full copies of records, even when a version is somewhere in the local cache, slowing things down and incurring cost. Some optimisation could have been applied here, I didn't find anything about it in the docs. Does this match others experience?
lytefm 5y ago
> At the end of the day, offline-first is a whole bunch of caching, and so you end up needing to deep dive into cache invalidation strategies which we all know is a hairy problem. Looks like you're not truly getting the concept of Offline-first apps then. You don't have or need a cache. You have a local database on the device that syncs up with the server, if online. Conflicts can occur, but modeling data well with a CouchDB/PouchDB setup (i.e. prevent conflicts by not modifying docs all the time, rahter create new ones) + having a simple time-stamp based heuristic can already be sufficient. Otherwise, using CRDT is an option.
- JamesSwift 5y ago
  > You don't have or need a cache. You have a local database on the device that syncs up with the server, if online. The source of truth is the server. Everything else (i.e. the local copy) is a snapshot of that, aka a cache. Its just that offline-first is _always_ a cache-first read, so you seem to think that this makes it not a cache any more, but a regular data store.
  - BackBlast 5y ago
    CRDT is really a multi-master model with eventual consistency. It's not merely a cache but also operates as a master and source of truth. With a well-built CRDT you can also skip the server and sync client-to-client where both have master copies. The tricky part is conflict resolution.
    - JamesSwift 5y ago
      Right, the peer-to-peer or anything with a concept of multi-master / majority is a different class of problem. I am speaking only to traditional client-server scenarios.
  - lytefm 5y ago
    > so you seem to think that this makes it not a cache any more, but a regular data store. Yes. In an true offline-first approach with CouchDB/PouchDB, the client-side database can be considered to be main data store and the Server-Side DB could just be a backup. Or it might not be needed at all/ might only be used to migrate from one device to another. I'd say whether it's a master-slave or Multi-master model depends on the conflict resolution strategy.
ssss11 5y ago
Your comment about an offline first native app being magical has just made me consider… are we in an age where it’s more common for developers to not have lived thru pc apps pre-internet/early internet? What you describe used to be the norm.
the_gipsy 5y ago
> Native apps dont deal with the same issues around storage, This is a synthetic problem on iOS, created by apple to push users away from webapps. It’s not only safari, apple has completely blocked any browser engines other than their own. Yes, a lot of commenters justify this by saying they want the web to be documents, and everything else native apps, that every page doesn’t have to be an SPA. For me it’s the opposite - I really don’t want to install another invasive app, just gimme a webapp. But sadly, apple crippled both storage and notifications enough to force developers AND users to use their appstore. This is lock-in, against the interest of their users, for their marketshare. The exact same tactics that MS was crucified for.
chadcmulligan 5y ago
> Native apps dont deal with the same issues around storage, and are actually _much_ more performant overall Its funny you have to say this - there's a whole generation of programmers who just think web first.

Trufa 5y ago

Yeah,I'm trying to implement an electron offline first app that syncs, there seems to readymade solution. Stuff like https://github.com/aerogear/offix https://github.com/aerogear/offix seem to be in the right direction of what I'm looking for but not nearly mature enough. I don't want to pu to much effort on the app so I would like something more or less ready made, preferably with graphql apis. Any suggestions welcome.

JamesSwift 5y ago
I would hesitate to use any client-server setup that isnt also your primary data store. So, if you have an "offline aware, multi-tenant" store like CouchDB but you still need to sync to the primary store which is SQL, then you lose a lot of context and awareness around conflict resolution. If you are going to eventually sync to the other store, I would say only use Couch on the frontend, and do the syncing/resolution from the perspective of the client, since the client knows what it was trying to do and how best to notify on conflict. The "better" option is to have an offline-aware client-server component (e.g. Realm) which is the primary store as well. This eliminates the sync and so all conflict resolution stays in the same system with well defined semantics.
- Trufa 5y ago
  Thanks for the recommendation, I'm a little bit skeptical of MongoDB, but realm looks nice, Firebase I'm a little reluctant on being so dependant on google, but otherwise looks good.
  - BackBlast 5y ago
    PouchDB works well as the primary client store and can sync to CouchDB in the cloud. IMHO this is one of the more mature combinations that gives you a ready client side document database that takes care of data replication for you.
    - WorldMaker 5y ago
      It's just unfortunate that running CouchDB in the cloud seems increasingly perilous. Since IBM acquired Cloudant, it's much tougher to run CouchDB "at scale PaaS" on any data center other than IBM's (for obvious reasons), whereas early Cloudant had robust Azure and AWS support for years. I wish Couchbase were more helpful practically than they try to present themselves theoretically. Even if their products weren't so expensive the impedance mismatch between their version of CouchDB's sync APIs and their own APIs seems to increase by the year, and is pretty noticeable in how different it works from a PouchDB standpoint and how easy it is to break sync. (Impedance mismatches in allowable database names and _id keys are huge on their own that have massive repercussions in application design.) Even CouchDB is not CouchDB anymore with impedance mismatches of its own between versions. On the one hand it's good that Cloudant upstreamed a lot of their cluster management tools directly into Apache CouchDB 2+ (even as they made their PaaS offering IBM Cloud only [or whatever it's name of the month is]), but huge architectural changes below the covers in CouchDB 3+ start to present their own sync issues akin to but distinct from Couchbase's (and even some of Cloudant's as they seem [?] to be diverging again into their own 2+ fork after all that work upstreaming stuff?). More than ever, Azure CosmosDB's focus on bare minimum MongoDB capability and not supporting anything like CouchDB sync, despite having close the same raw ingredients (Cosmos' change feed looks a lot more like Couch's, but is just missing a couple subtle things to make it directly and immediately useful for Couch replication) seems like a "CouchDB is dead and not worth supporting" signal from Microsoft. Unfortunately, I think PouchDB <=> CouchDB replication has past the "mature" point to the "decrepit" and "falling apart" stage, maybe to the point of "evolutionary dead end" if I'm feeling strongly pessimistic enough, and I've been for years trying to figure what to replace it with.
firebase-user 5y ago
You should try using Firebase. It handles the data-sync for you, and it applies all updates locally first so that your app feels snappy.
- danuker 5y ago
  Ah yes. Convenience is how Big Tech gets all data to pass through it.
- JamesSwift 5y ago
  I mentioned Realm because thats what Im familiar with but yeah Firebase is also a good choice if you want a client-server setup.
- code-is-code 5y ago
  Firebase works until you want a different conflict resolution then last-write-wins. It is also only offline first if the user is authenticated. Otherwise you need a connection to the servers before using the local state.
- jamil7 5y ago
  Firebase is offline-resistant more so than offline-first I’d argue.
- LAC-Tech 5y ago
  Last write wins is not handling data sync, it's washing your hands of it. Your users will be frustrated.
- davidzweig 5y ago
  The (JS) frontend SDK can be set to either online or offline mode. When in online mode, it seems to insist on downloading full copies of records, even when a version is somewhere in the local cache, slowing things down and incurring cost. Some optimisation could have been applied here, I didn't find anything about it in the docs. Does this match others experience?

lytefm 5y ago

> At the end of the day, offline-first is a whole bunch of caching, and so you end up needing to deep dive into cache invalidation strategies which we all know is a hairy problem. Looks like you're not truly getting the concept of Offline-first apps then. You don't have or need a cache. You have a local database on the device that syncs up with the server, if online. Conflicts can occur, but modeling data well with a CouchDB/PouchDB setup (i.e. prevent conflicts by not modifying docs all the time, rahter create new ones) + having a simple time-stamp based heuristic can already be sufficient. Otherwise, using CRDT is an option.

JamesSwift 5y ago
> You don't have or need a cache. You have a local database on the device that syncs up with the server, if online. The source of truth is the server. Everything else (i.e. the local copy) is a snapshot of that, aka a cache. Its just that offline-first is _always_ a cache-first read, so you seem to think that this makes it not a cache any more, but a regular data store.
- BackBlast 5y ago
  CRDT is really a multi-master model with eventual consistency. It's not merely a cache but also operates as a master and source of truth. With a well-built CRDT you can also skip the server and sync client-to-client where both have master copies. The tricky part is conflict resolution.
  - JamesSwift 5y ago
    Right, the peer-to-peer or anything with a concept of multi-master / majority is a different class of problem. I am speaking only to traditional client-server scenarios.
- lytefm 5y ago
  > so you seem to think that this makes it not a cache any more, but a regular data store. Yes. In an true offline-first approach with CouchDB/PouchDB, the client-side database can be considered to be main data store and the Server-Side DB could just be a backup. Or it might not be needed at all/ might only be used to migrate from one device to another. I'd say whether it's a master-slave or Multi-master model depends on the conflict resolution strategy.

ssss11 5y ago

Your comment about an offline first native app being magical has just made me consider… are we in an age where it’s more common for developers to not have lived thru pc apps pre-internet/early internet? What you describe used to be the norm.

the_gipsy 5y ago

> Native apps dont deal with the same issues around storage, This is a synthetic problem on iOS, created by apple to push users away from webapps. It’s not only safari, apple has completely blocked any browser engines other than their own. Yes, a lot of commenters justify this by saying they want the web to be documents, and everything else native apps, that every page doesn’t have to be an SPA. For me it’s the opposite - I really don’t want to install another invasive app, just gimme a webapp. But sadly, apple crippled both storage and notifications enough to force developers AND users to use their appstore. This is lock-in, against the interest of their users, for their marketshare. The exact same tactics that MS was crucified for.

chadcmulligan 5y ago

> Native apps dont deal with the same issues around storage, and are actually _much_ more performant overall Its funny you have to say this - there's a whole generation of programmers who just think web first.