6 ms·

I still haven't found the "holy grail architecture" for offline-first with backend sync where the backend isn't just a simple data store but also has business l

by zorr 5y ago

I still haven't found the "holy grail architecture" for offline-first with backend sync where the backend isn't just a simple data store but also has business logic and interacts with external systems. Doing offline-first well implies that the app has a (sqlite or similar) local database, does work on that database and periodically syncs the changes to the backend. This means you have N+1 databases to keep in sync (with N=number of clients). Essentially distributed database replication where clients can go offline at any time. Potentially with different tech (for example sqlite on the client and postgres on the backend). When the backend isn't very smart it's not too hard, you can just encapsulate any state-changing action into an event, when offline process the events locally + cache the chain of events, when online send the chain to the backend and have the backend apply the events one by one. Periodically sync new events from the backend and apply them locally to stay in sync. This is what classic Todo apps like OmniFocus do. The problems start when the backend is smarter and also applies business logic that generates new events (for example enriching new entities with data from external systems, or adding new tasks to a timeline etc). Obviously the new server-generated events are only available later when the client comes back online. When trying to make the offline experience as feature-rich as possible I always end up duplicating almost all of the backend logic into the clients as well. And in that case, what is even the point of having a smart backend.

aboodman 5y ago
You should take a look at my project, Replicache: replicache.dev. While it is true that you have to duplicate the mutations in the basic setup, you do not have to share the querying/reading code as it lives primarily on the client. Also, if your backend happens to be javascript/typescript, then you can share the majority of the mutation code between client and server and the result is quite sweet.
- zorr 5y ago
  When implementing something like this I indeed prefer to share as much code as possible between client/server. My daily environment is mostly JVM-based (desktop/android/server) so your project is probably not a great fit for me but I'm definitely going to look into it for some inspiration.
drewjoh 5y ago
Check out WatermelonDB: https://nozbe.github.io/WatermelonDB https://nozbe.github.io/WatermelonDB
kybernetikos 5y ago
I've been looking for something similar and found a lot of approaches, spanning CRDTs, reactive databases, etc all of which look really promising. For me one of the missing pieces is situations where some parts of the object graph should not be visible or editable by certain users. That sort of thing comes up the whole time and can be pretty complex yet is often just handwaved away.
- dharmaturtle 5y ago
  > one of the missing pieces is situations where some parts of the object graph should not be visible or editable by certain users. That sort of thing comes up the whole time and can be pretty complex yet is often just handwaved away. I discovered this problem in my domain about two months ago. My solution was to split my Aggregate Root into smaller pieces, to use DDD terminology. In my domain, each user/client owns their own data. However they may choose to publish that data and have it be publicly visible so others may view/comment/copy/pull-request it. It's basically Github for flashcards. So, I have an Aggregate Root for publicly visible flashcards, and another Aggregate Root for a user's personal flashcards. It helps that there are distinct behaviors for each Aggregate Root - there's no real point to leaving a comment on a personal flashcard, and there's no real point to "studying" a public flashcard (because that implies logging your study history to the public card). It does mean, however, that there needs to be a translation layer - it should be possible to convert a private flashcard to a publicly visible one, and it should also be possible to copy a public flashcard to your own personal collection. Obviously this is very domain specific.
JamesSwift 5y ago
Ive found the same. There is no silver bullet with offline-first. Its extremely product / architecture specific. But I do lean heavily on the side of offline-first being a _much_ better experience overall.
dharmaturtle 5y ago
Seems like you're describing event sourcing. I'm building an offline-first app and doing pretty much what you're describing. > I always end up duplicating almost all of the backend logic into the clients as well Yep, this is a pain I feel acutely. I'm using dotnet (C#/F#) because it allows me to ship & run DLLs on the browser with Blazor, leading to significantly less duplicate code. F# can transpile to Javascript with Fable (F# + Babel), so that's also an option. I haven't fully vetted Blazor yet, but it seems good. Clojure can also run on the server and on the browser with ClojureScript. The only other option I see is going fullstack Javascript, and I hate Javascript.
- zorr 5y ago
  > Seems like you're describing event sourcing. Yes it looks like event sourcing. But most of the classic event sourcing implementations I've seen are mostly backend only. The project I'm currently hacking on has an MQTT broker that is used by both clients and backend and the events can come from anywhere. For example when a client sends a `location_updated` event, the backend reverse-geocodes this into an address and possibly sends out `place_entered` events, or sends out notifications for Tasks that are now relevant on the new location, or "completes" a "go to location X" task resulting in a `task_completed` event. Enabling all this in an offline-first paradigm is hard and requires a lot of duplication, I am very close to just saying screw this and requiring network connectivity for most of the features.
- JamesSwift 5y ago
  .net is also my choice for this. Xamarin on mobile and hopefully blazor on web in the future. Kotlin is the other contender to keep an eye on in the future (Kotlin Native and Kotlin Multiplatform)
nonameiguess 5y ago
For all the crap the DoD gets, this has been a solved problem forever in the military. Even putting aside technical things like the Blue Force Tracker or ABMS that present a global view disseminated to users in the field who also feed information back to the distributed data store, it is simply expected and accepted that data will not be consistent in the presence of sparse network connectivity, which you will inevitably sometimes have, especially in the older days when relying on radio mesh networks or even further back before this was primarily a technological problem when headquarters communicated with forward units via horseback messenger. If the decision point is more critical to get right, you wait until you have reasonable assurance your information is accurate and consistent. If the decision point is more critical to be moved past quickly, then you act on the last known state even though it may no longer be accurate. If it's most important that all units be on the same page, then you act on the last known agreed upon plan, even if some units have newer information. They don't update until they get positive confirmation from headquarters that every unit has received the new information. Heck, as much as I don't like them, even Facebook got this right last I knew when I still used their app years ago. It never required a network connection. If you weren't actively receiving updates, it just showed you the cached last known feed, accepting that it wasn't up to date. And if you tried to post something, it would just cache that too and wait to send it. They didn't invent eventual consistency, either. It's been a basic operating principle of distributed organizations, especially armies, for thousands of years.
the_arun 5y ago
What if N clients use N versions of the client software?
seph-reed 5y ago
I have one I've been working on for a few years. It's based off something I call "source derivation".... It has data sources where things get stored, repos that handle syncing and pulling data, and ents which represent the data but add functions to it.
beckman466 5y ago
> I still haven't found the "holy grail architecture" for offline-first with backend sync where the backend isn't just a simple data store but also has business logic and interacts with external systems. you might like the architecture of holo-chain (stupid name, imo), it's still mostly going under the radar (also because they chose to refactor their library from go to rust to improve security): "Holochain combines ideas from BitTorrent and Git, along with cryptographic signatures, peer validation, and gossip. Holochain is an open source framework for building fully distributed, peer-to-peer applications. [It's] purpose is to enable humans to interact with each other by mutual-consent to a shared set of rules, without relying on any authority to dictate or unilaterally change those rules. Peer-to-peer interaction means you own and control your data, with no intermediary" good dev intro: https://medium.com/holochain/holochain-reinventing-applications-d2ac1e4f25ef https://medium.com/holochain/holochain-reinventing-applicati... "Holochain is [...] like a decentralized Ruby on Rails, a toolkit that produces stand-alone programs but for serverless high-profile applications."
- Powersource 5y ago
  Haha, didn't think I'd run into you here tonight
  - beckman466 5y ago
    hey there friend
- JamesSwift 5y ago
  That sounds like it solves part of the sync problem, but not conflict resolution or offline business rules in a general way.
  - beckman466 5y ago
    > does not solve offline business rules in a general way hmm no i think it does. the 'Resilience and availability' section is relevant for your question: https://developer.holochain.org/concepts/4_dht/ https://developer.holochain.org/concepts/4_dht/ holochain is completely distributed, so there is no 'offline' and 'online' because there is no third party. it sounds weird writing that but i'm hoping it might challenge you to dig a bit deeper into the docs, because it's there. if not, please come back and let me know so i can pass on feedback about the docs that were unclear or unsatisfactory for you. > does not solve conflict resolution from the docs: "Holochain DNAs specify validation rules for every type of entry or link. This empowers agents to check the integrity of the data they see. When called upon to validate data, it allows them to identify corrupt peers and publish a warrant against them. [...] (the DNA is simply a collection of functions for creating, accessing, and validating data.)" https://developer.holochain.org/concepts/7_validation/ https://developer.holochain.org/concepts/7_validation/
deleted 5y ago
[deleted]

by zorr 5y ago

aboodman 5y ago

You should take a look at my project, Replicache: replicache.dev. While it is true that you have to duplicate the mutations in the basic setup, you do not have to share the querying/reading code as it lives primarily on the client. Also, if your backend happens to be javascript/typescript, then you can share the majority of the mutation code between client and server and the result is quite sweet.

zorr 5y ago
When implementing something like this I indeed prefer to share as much code as possible between client/server. My daily environment is mostly JVM-based (desktop/android/server) so your project is probably not a great fit for me but I'm definitely going to look into it for some inspiration.

drewjoh 5y ago

Check out WatermelonDB: https://nozbe.github.io/WatermelonDB https://nozbe.github.io/WatermelonDB

kybernetikos 5y ago

I've been looking for something similar and found a lot of approaches, spanning CRDTs, reactive databases, etc all of which look really promising. For me one of the missing pieces is situations where some parts of the object graph should not be visible or editable by certain users. That sort of thing comes up the whole time and can be pretty complex yet is often just handwaved away.

dharmaturtle 5y ago
> one of the missing pieces is situations where some parts of the object graph should not be visible or editable by certain users. That sort of thing comes up the whole time and can be pretty complex yet is often just handwaved away. I discovered this problem in my domain about two months ago. My solution was to split my Aggregate Root into smaller pieces, to use DDD terminology. In my domain, each user/client owns their own data. However they may choose to publish that data and have it be publicly visible so others may view/comment/copy/pull-request it. It's basically Github for flashcards. So, I have an Aggregate Root for publicly visible flashcards, and another Aggregate Root for a user's personal flashcards. It helps that there are distinct behaviors for each Aggregate Root - there's no real point to leaving a comment on a personal flashcard, and there's no real point to "studying" a public flashcard (because that implies logging your study history to the public card). It does mean, however, that there needs to be a translation layer - it should be possible to convert a private flashcard to a publicly visible one, and it should also be possible to copy a public flashcard to your own personal collection. Obviously this is very domain specific.

JamesSwift 5y ago

Ive found the same. There is no silver bullet with offline-first. Its extremely product / architecture specific. But I do lean heavily on the side of offline-first being a _much_ better experience overall.

dharmaturtle 5y ago

Seems like you're describing event sourcing. I'm building an offline-first app and doing pretty much what you're describing. > I always end up duplicating almost all of the backend logic into the clients as well Yep, this is a pain I feel acutely. I'm using dotnet (C#/F#) because it allows me to ship & run DLLs on the browser with Blazor, leading to significantly less duplicate code. F# can transpile to Javascript with Fable (F# + Babel), so that's also an option. I haven't fully vetted Blazor yet, but it seems good. Clojure can also run on the server and on the browser with ClojureScript. The only other option I see is going fullstack Javascript, and I hate Javascript.

zorr 5y ago
> Seems like you're describing event sourcing. Yes it looks like event sourcing. But most of the classic event sourcing implementations I've seen are mostly backend only. The project I'm currently hacking on has an MQTT broker that is used by both clients and backend and the events can come from anywhere. For example when a client sends a `location_updated` event, the backend reverse-geocodes this into an address and possibly sends out `place_entered` events, or sends out notifications for Tasks that are now relevant on the new location, or "completes" a "go to location X" task resulting in a `task_completed` event. Enabling all this in an offline-first paradigm is hard and requires a lot of duplication, I am very close to just saying screw this and requiring network connectivity for most of the features.
JamesSwift 5y ago
.net is also my choice for this. Xamarin on mobile and hopefully blazor on web in the future. Kotlin is the other contender to keep an eye on in the future (Kotlin Native and Kotlin Multiplatform)

nonameiguess 5y ago

For all the crap the DoD gets, this has been a solved problem forever in the military. Even putting aside technical things like the Blue Force Tracker or ABMS that present a global view disseminated to users in the field who also feed information back to the distributed data store, it is simply expected and accepted that data will not be consistent in the presence of sparse network connectivity, which you will inevitably sometimes have, especially in the older days when relying on radio mesh networks or even further back before this was primarily a technological problem when headquarters communicated with forward units via horseback messenger. If the decision point is more critical to get right, you wait until you have reasonable assurance your information is accurate and consistent. If the decision point is more critical to be moved past quickly, then you act on the last known state even though it may no longer be accurate. If it's most important that all units be on the same page, then you act on the last known agreed upon plan, even if some units have newer information. They don't update until they get positive confirmation from headquarters that every unit has received the new information. Heck, as much as I don't like them, even Facebook got this right last I knew when I still used their app years ago. It never required a network connection. If you weren't actively receiving updates, it just showed you the cached last known feed, accepting that it wasn't up to date. And if you tried to post something, it would just cache that too and wait to send it. They didn't invent eventual consistency, either. It's been a basic operating principle of distributed organizations, especially armies, for thousands of years.

the_arun 5y ago

What if N clients use N versions of the client software?

seph-reed 5y ago

I have one I've been working on for a few years. It's based off something I call "source derivation".... It has data sources where things get stored, repos that handle syncing and pulling data, and ents which represent the data but add functions to it.

beckman466 5y ago

> I still haven't found the "holy grail architecture" for offline-first with backend sync where the backend isn't just a simple data store but also has business logic and interacts with external systems. you might like the architecture of holo-chain (stupid name, imo), it's still mostly going under the radar (also because they chose to refactor their library from go to rust to improve security): "Holochain combines ideas from BitTorrent and Git, along with cryptographic signatures, peer validation, and gossip. Holochain is an open source framework for building fully distributed, peer-to-peer applications. [It's] purpose is to enable humans to interact with each other by mutual-consent to a shared set of rules, without relying on any authority to dictate or unilaterally change those rules. Peer-to-peer interaction means you own and control your data, with no intermediary" good dev intro: https://medium.com/holochain/holochain-reinventing-applications-d2ac1e4f25ef https://medium.com/holochain/holochain-reinventing-applicati... "Holochain is [...] like a decentralized Ruby on Rails, a toolkit that produces stand-alone programs but for serverless high-profile applications."

Powersource 5y ago
Haha, didn't think I'd run into you here tonight
- beckman466 5y ago
  hey there friend
JamesSwift 5y ago
That sounds like it solves part of the sync problem, but not conflict resolution or offline business rules in a general way.
- beckman466 5y ago
  > does not solve offline business rules in a general way hmm no i think it does. the 'Resilience and availability' section is relevant for your question: https://developer.holochain.org/concepts/4_dht/ https://developer.holochain.org/concepts/4_dht/ holochain is completely distributed, so there is no 'offline' and 'online' because there is no third party. it sounds weird writing that but i'm hoping it might challenge you to dig a bit deeper into the docs, because it's there. if not, please come back and let me know so i can pass on feedback about the docs that were unclear or unsatisfactory for you. > does not solve conflict resolution from the docs: "Holochain DNAs specify validation rules for every type of entry or link. This empowers agents to check the integrity of the data they see. When called upon to validate data, it allows them to identify corrupt peers and publish a warrant against them. [...] (the DNA is simply a collection of functions for creating, accessing, and validating data.)" https://developer.holochain.org/concepts/7_validation/ https://developer.holochain.org/concepts/7_validation/

deleted 5y ago

[deleted]