6 ms·

> You’ll get your data synced for you How does this happen without an interface for conflict resolution? That's the hard part.

Sammi 2y ago
All this recent hype about sync engines and local first applications completely disregards conflict resolution. It's the reason why syncing isn't mainstream already and it isn't solved and arguably cannot be. Imagine if git just on its own picked what to keep and what to throw away when there's a conflict. You fundamentally need the user to make the choice.
- porridgeraisin 2y ago
  Precisely. The hype articles write all about the journey to The Wall, and then leave out the bit where you smash headfirst into it.
- lifty 2y ago
  Very good point. The local-sync ecosystem is still in a young phase, and conflict resolution hasn't been tackled or solved yet. Most systems have a |last write wins" approach.
- sgt 2y ago
  > All this recent hype about sync engines and local first applications completely disregards conflict resolution. Not really true though. I've used a couple of local sync engines, one internally built and another one which is both commercial and now open source called PowerSync[1]. Conflict resolution is definitely on the agenda, and a developer is definitely going to be mindful of conflicts when designing the application. [1] https://www.powersync.com/ https://www.powersync.com/
  - Sammi 2y ago
    My unfortunate point is that the dev cannot know what the user is doing, and so cannot in principle know what choice to make on behalf of the user in case of a conflict. This is not a code problem. It cannot be solved with code.
    - sgt 2y ago
      I've found that in almost all cases - the latest update "wins" strategy is fine. You could have two sessions working with conventional API calls and still have a conflict. As a dev you need to restrict what the user can do.
- Jyaif 2y ago
  > All this recent hype about sync engines and local first applications completely disregards conflict resolution The main concern of sync engines is precisely the conflict resolution! Everything else is simple in comparison. The good news is that under some circumstances it is possible to solve conflicts without user intervention. The simplest example is a counter that can only be incremented. More advanced data structures automatically solving conflicts exists, for example solving conflicts for strings exists, and those are good enough for a text editor. I agree that there will be conflicts that are resolved in a way that yields non-sensical text, for example if there are 2 edits of the sentence "One cat": One cat => Two cats One cat => One dog The resulting merge may be something like "Two cats dog". Something else (the user, an LLM...) will then have to fix it. But that's totally OK, because in practice this will happen extremely rarely, only when the user would have been offline for a long time. That user will be happy to have been able to work offline, largely compensating the fact that they have to proof read the text again.
  - SkiFire13 2y ago
    This doesn't "solve" conflict resolution, it just picks one of the possible answers and then doesn't care whether it was the correct one or not. It can be acceptable for some usecases, but not for others where you're still concerned about stuff that happens "extremely rately" and is not under your direct control. > Something else (the user, an LLM...) will then have to fix it. This assumes that user/llm knows the conflict was automatically solved and might need to be fixed, so the conflict is still there! You just made the manual part delayed and non-mandatory, but if you want correctness it will still have to be there.
  - brulard 2y ago
    > in practice this will happen extremely rarely, only when the user would have been offline for a long time. I don't think it would happen "extremely rarely". Drops in connectivity happen a lot, especially on cellular connection and this can absolutely happen a lot for some applications. Especially when talking about "offline first" apps.
    - Jyaif 2y ago
- aboodman 2y ago
  Zero (zerosync.dev) uses transactional conflict resolution, which is what our prior product Replicache and Reflect both used. It is very similar to what multiplayer games have done for decades. It is described here: https://rocicorp.dev/blog/ready-player-two https://rocicorp.dev/blog/ready-player-two It works really well and we and our customers have found it to be quite general. It allows you to run an arbitrary transaction on the sever side to decide what to do in case of conflicts. It is the software equivalent of git asking the user what to do. Zero asks your code what to do. But it asks it in the form of the question "please run the function named x with these inputs on the current backend db state". Which is a much more ergonomic way to ask it than "please do a 3-way merge between these three states". Conflict resolution is not the reason why there has not been a general-purpose sync engine. None of our customers have ~ever complained about conflict resolution. The reason there has not been a general-purpose sync engine is actually on the read side: - Previous sync engines really want you to sync all data. This is impractical for most apps. - Previous sync engines do not have practical approaches to permissions. These problems are being solved in next generation of sync engines. For more on this, I talk about it some here: https://www.youtube.com/watch?v=rqOUgqsWvbw https://www.youtube.com/watch?v=rqOUgqsWvbw
  - probabletrain 2y ago
    I think with good presence (being able to see what other users are doing) and an app that isn't used offline, conflicts are essentially not a problem. As long as whatever is resolving the conflicts resolves them in a way that doesn't break the app, e.g. making sure there aren't cycles in some multiplayer app with a tree datastructure. Sounds like Zero has the right idea here, I'll build something on it imminently to try it out.
    - Sammi 2y ago
      Agree that if you don't have offline support, then conflict resolution is such a minor issue that you can just do "last write wins" and call it a day.
  - probabletrain 2y ago
    > Previous sync engines really want you to sync all data Linear had to do all sorts of shenanigans to be able to sync all data, for orgs with lots of it – there's a talk on that here: https://www.youtube.com/watch?v=Wo2m3jaJixU&t=1473s https://www.youtube.com/watch?v=Wo2m3jaJixU&t=1473s
- jamil7 2y ago
  > All this recent hype about sync engines and local first applications Kind of but only really in the web world, it was the default on desktop for a long time and is pretty common on mobile.
- deleted 2y ago
  [deleted]
phito 2y ago
Right, first thing I did after opening the article is CTRL-F'ing for conflict, and got zero result. How are they not talking about the only real problem about the local-first approach? The rest is just boiler plate code.
tonsky 2y ago
Ah, no. Not really. People sometimes think about conflict resolution as a problem that needs to be solved. But it’s not solvable, not really. It’s part of the domain, it’s not going anywhere, it’s irreducible complexity. You _will_ have conflicts (because your app is distributed and there are concurrent writes). They will happen on semantic level, so only you (app developer) _will_ be able to solve them. Database (or any other magical tool) can’t do it for you. Another misconception is that conflict resolution needs to be “solved” perfectly before any progress can be made. That is not true as well. You might have unhandled conflicts in your system and still have a working, useful, successful product. Conflicts might be rare, insignificant, or people (your users) will just correct for/work around them. I am not saying “drop data on the floor”, of course, if you can help it. But try not to overthink it, either.
- DaiPlusPlus 2y ago
  > But it’s not solvable, not really. It’s part of the domain, it’s not going anywhere, it’s irreducible complexity. You _will_ have conflicts (because your app is distributed and there are concurrent writes). [...] Another misconception is that conflict resolution needs to be “solved” perfectly before any progress can be made. That is not true as well. You might have unhandled conflicts in your system and still have a working, useful, successful product. Conflicts might be rare, insignificant, or people (your users) will just correct for/work around them. I can't speak for whatever application-level problems you were trying to solve, but many problem-cases can be massaged into being conflict-free by adding constraints (or rather: discovering constraints inherent in the business-domain you can use). For example (and the best example, too) is to use an append-only logical model: then the synchronization problem reduces down to merge-sort. Another kind of constraint might be to simply disallow "edit" access to local data when working-offline (without a prior lock or lease being taken) but still allowing "create". > Database (or any other magical tool) can’t do it for you. Yes-and-no. While I'm no fan of CORBA and COM+ (...or SOAP, or WS-OhGodMakeItStop), but being "enterprise-y" it meant they brought distributed-transactions to any application, and that includes RDBMS-mediated distributed transactions (let's agree, an RDBMS is in a far greater position to be a better canonical transaction-server than an application-server running in-front of it). For distributed systems needing transient distributed locks to prevent conflicts in the first place (so only used by interactive users in the same LAN, really) this worked just-as-well as a local-only solution - and make it fault-tolerant too. ...so it is unfortunate that with the (absolutely justified) back-to-basics approach with REST[1] that we lose built-in support for distributed transactions (even some of the more useful and legitimate parts of WebDAV (and so, piggy-backing on our web-servers' built-in support for WebDAV verbs) seem to be going-away) - this all raises the barrier-to-entry for doing distributed-transactions _right_, which means the next set of college-hires won't have been exposed to it, which means it won't be a standard expected feature in the next major internal application they'll write for your org, which means you'll either have a race-condition impacting a multi-billion-dollar business thing that no-one knows how to fix or more likely, just a crappy UX where you have to tell your users not to reload the page too quickly "just in case". Yes, I see advisories like that in the Zendesk pages of the next line-of-business SaaS you'll be voluntold to integrate into your org. (I think today, the "best" way to handle distributed-locking between interactive-users in a web-app would necessitate using a ServiceWorker using WebRTC, SSE, or a highly-reliable WebSocket - which itself is a load of work right there - and don't forget to do all your JS feature-checks because eventually someone will try to use your app on an old Safari edition because they want to keep on using their vintage Mac) - or anyone using Incognito mode, _gah_. [1]: https://devblast.com/b/calling-your-web-api-restful-youre-doing-it-wrong https://devblast.com/b/calling-your-web-api-restful-youre-do...
avodonosov 2y ago
They elaborate on the conflicts in the "80/20 for Multiplayer" section of this essay: https://www.instantdb.com/essays/next_firebase https://www.instantdb.com/essays/next_firebase (make sure to also read the footnote [28] there).

Sammi 2y ago

All this recent hype about sync engines and local first applications completely disregards conflict resolution. It's the reason why syncing isn't mainstream already and it isn't solved and arguably cannot be. Imagine if git just on its own picked what to keep and what to throw away when there's a conflict. You fundamentally need the user to make the choice.

porridgeraisin 2y ago
Precisely. The hype articles write all about the journey to The Wall, and then leave out the bit where you smash headfirst into it.
lifty 2y ago
Very good point. The local-sync ecosystem is still in a young phase, and conflict resolution hasn't been tackled or solved yet. Most systems have a |last write wins" approach.
sgt 2y ago
> All this recent hype about sync engines and local first applications completely disregards conflict resolution. Not really true though. I've used a couple of local sync engines, one internally built and another one which is both commercial and now open source called PowerSync[1]. Conflict resolution is definitely on the agenda, and a developer is definitely going to be mindful of conflicts when designing the application. [1] https://www.powersync.com/ https://www.powersync.com/
- Sammi 2y ago
  My unfortunate point is that the dev cannot know what the user is doing, and so cannot in principle know what choice to make on behalf of the user in case of a conflict. This is not a code problem. It cannot be solved with code.
  - sgt 2y ago
    I've found that in almost all cases - the latest update "wins" strategy is fine. You could have two sessions working with conventional API calls and still have a conflict. As a dev you need to restrict what the user can do.
Jyaif 2y ago
> All this recent hype about sync engines and local first applications completely disregards conflict resolution The main concern of sync engines is precisely the conflict resolution! Everything else is simple in comparison. The good news is that under some circumstances it is possible to solve conflicts without user intervention. The simplest example is a counter that can only be incremented. More advanced data structures automatically solving conflicts exists, for example solving conflicts for strings exists, and those are good enough for a text editor. I agree that there will be conflicts that are resolved in a way that yields non-sensical text, for example if there are 2 edits of the sentence "One cat": One cat => Two cats One cat => One dog The resulting merge may be something like "Two cats dog". Something else (the user, an LLM...) will then have to fix it. But that's totally OK, because in practice this will happen extremely rarely, only when the user would have been offline for a long time. That user will be happy to have been able to work offline, largely compensating the fact that they have to proof read the text again.
- SkiFire13 2y ago
  This doesn't "solve" conflict resolution, it just picks one of the possible answers and then doesn't care whether it was the correct one or not. It can be acceptable for some usecases, but not for others where you're still concerned about stuff that happens "extremely rately" and is not under your direct control. > Something else (the user, an LLM...) will then have to fix it. This assumes that user/llm knows the conflict was automatically solved and might need to be fixed, so the conflict is still there! You just made the manual part delayed and non-mandatory, but if you want correctness it will still have to be there.
- brulard 2y ago
  > in practice this will happen extremely rarely, only when the user would have been offline for a long time. I don't think it would happen "extremely rarely". Drops in connectivity happen a lot, especially on cellular connection and this can absolutely happen a lot for some applications. Especially when talking about "offline first" apps.
  - Jyaif 2y ago
aboodman 2y ago
Zero (zerosync.dev) uses transactional conflict resolution, which is what our prior product Replicache and Reflect both used. It is very similar to what multiplayer games have done for decades. It is described here: https://rocicorp.dev/blog/ready-player-two https://rocicorp.dev/blog/ready-player-two It works really well and we and our customers have found it to be quite general. It allows you to run an arbitrary transaction on the sever side to decide what to do in case of conflicts. It is the software equivalent of git asking the user what to do. Zero asks your code what to do. But it asks it in the form of the question "please run the function named x with these inputs on the current backend db state". Which is a much more ergonomic way to ask it than "please do a 3-way merge between these three states". Conflict resolution is not the reason why there has not been a general-purpose sync engine. None of our customers have ~ever complained about conflict resolution. The reason there has not been a general-purpose sync engine is actually on the read side: - Previous sync engines really want you to sync all data. This is impractical for most apps. - Previous sync engines do not have practical approaches to permissions. These problems are being solved in next generation of sync engines. For more on this, I talk about it some here: https://www.youtube.com/watch?v=rqOUgqsWvbw https://www.youtube.com/watch?v=rqOUgqsWvbw
- probabletrain 2y ago
  I think with good presence (being able to see what other users are doing) and an app that isn't used offline, conflicts are essentially not a problem. As long as whatever is resolving the conflicts resolves them in a way that doesn't break the app, e.g. making sure there aren't cycles in some multiplayer app with a tree datastructure. Sounds like Zero has the right idea here, I'll build something on it imminently to try it out.
  - Sammi 2y ago
    Agree that if you don't have offline support, then conflict resolution is such a minor issue that you can just do "last write wins" and call it a day.
- probabletrain 2y ago
  > Previous sync engines really want you to sync all data Linear had to do all sorts of shenanigans to be able to sync all data, for orgs with lots of it – there's a talk on that here: https://www.youtube.com/watch?v=Wo2m3jaJixU&t=1473s https://www.youtube.com/watch?v=Wo2m3jaJixU&t=1473s
jamil7 2y ago
> All this recent hype about sync engines and local first applications Kind of but only really in the web world, it was the default on desktop for a long time and is pretty common on mobile.
deleted 2y ago
[deleted]

phito 2y ago

Right, first thing I did after opening the article is CTRL-F'ing for conflict, and got zero result. How are they not talking about the only real problem about the local-first approach? The rest is just boiler plate code.

tonsky 2y ago

Ah, no. Not really. People sometimes think about conflict resolution as a problem that needs to be solved. But it’s not solvable, not really. It’s part of the domain, it’s not going anywhere, it’s irreducible complexity. You _will_ have conflicts (because your app is distributed and there are concurrent writes). They will happen on semantic level, so only you (app developer) _will_ be able to solve them. Database (or any other magical tool) can’t do it for you. Another misconception is that conflict resolution needs to be “solved” perfectly before any progress can be made. That is not true as well. You might have unhandled conflicts in your system and still have a working, useful, successful product. Conflicts might be rare, insignificant, or people (your users) will just correct for/work around them. I am not saying “drop data on the floor”, of course, if you can help it. But try not to overthink it, either.

DaiPlusPlus 2y ago
> But it’s not solvable, not really. It’s part of the domain, it’s not going anywhere, it’s irreducible complexity. You _will_ have conflicts (because your app is distributed and there are concurrent writes). [...] Another misconception is that conflict resolution needs to be “solved” perfectly before any progress can be made. That is not true as well. You might have unhandled conflicts in your system and still have a working, useful, successful product. Conflicts might be rare, insignificant, or people (your users) will just correct for/work around them. I can't speak for whatever application-level problems you were trying to solve, but many problem-cases can be massaged into being conflict-free by adding constraints (or rather: discovering constraints inherent in the business-domain you can use). For example (and the best example, too) is to use an append-only logical model: then the synchronization problem reduces down to merge-sort. Another kind of constraint might be to simply disallow "edit" access to local data when working-offline (without a prior lock or lease being taken) but still allowing "create". > Database (or any other magical tool) can’t do it for you. Yes-and-no. While I'm no fan of CORBA and COM+ (...or SOAP, or WS-OhGodMakeItStop), but being "enterprise-y" it meant they brought distributed-transactions to any application, and that includes RDBMS-mediated distributed transactions (let's agree, an RDBMS is in a far greater position to be a better canonical transaction-server than an application-server running in-front of it). For distributed systems needing transient distributed locks to prevent conflicts in the first place (so only used by interactive users in the same LAN, really) this worked just-as-well as a local-only solution - and make it fault-tolerant too. ...so it is unfortunate that with the (absolutely justified) back-to-basics approach with REST[1] that we lose built-in support for distributed transactions (even some of the more useful and legitimate parts of WebDAV (and so, piggy-backing on our web-servers' built-in support for WebDAV verbs) seem to be going-away) - this all raises the barrier-to-entry for doing distributed-transactions _right_, which means the next set of college-hires won't have been exposed to it, which means it won't be a standard expected feature in the next major internal application they'll write for your org, which means you'll either have a race-condition impacting a multi-billion-dollar business thing that no-one knows how to fix or more likely, just a crappy UX where you have to tell your users not to reload the page too quickly "just in case". Yes, I see advisories like that in the Zendesk pages of the next line-of-business SaaS you'll be voluntold to integrate into your org. (I think today, the "best" way to handle distributed-locking between interactive-users in a web-app would necessitate using a ServiceWorker using WebRTC, SSE, or a highly-reliable WebSocket - which itself is a load of work right there - and don't forget to do all your JS feature-checks because eventually someone will try to use your app on an old Safari edition because they want to keep on using their vintage Mac) - or anyone using Incognito mode, _gah_. [1]: https://devblast.com/b/calling-your-web-api-restful-youre-doing-it-wrong https://devblast.com/b/calling-your-web-api-restful-youre-do...

avodonosov 2y ago

They elaborate on the conflicts in the "80/20 for Multiplayer" section of this essay: https://www.instantdb.com/essays/next_firebase https://www.instantdb.com/essays/next_firebase (make sure to also read the footnote [28] there).