4 ms·

All this recent hype about sync engines and local first applications completely disregards conflict resolution. It's the reason why syncing isn't mainstream alr

by Sammi 2y ago

All this recent hype about sync engines and local first applications completely disregards conflict resolution. It's the reason why syncing isn't mainstream already and it isn't solved and arguably cannot be. Imagine if git just on its own picked what to keep and what to throw away when there's a conflict. You fundamentally need the user to make the choice.

porridgeraisin 2y ago
Precisely. The hype articles write all about the journey to The Wall, and then leave out the bit where you smash headfirst into it.
lifty 2y ago
Very good point. The local-sync ecosystem is still in a young phase, and conflict resolution hasn't been tackled or solved yet. Most systems have a |last write wins" approach.
sgt 2y ago
> All this recent hype about sync engines and local first applications completely disregards conflict resolution. Not really true though. I've used a couple of local sync engines, one internally built and another one which is both commercial and now open source called PowerSync[1]. Conflict resolution is definitely on the agenda, and a developer is definitely going to be mindful of conflicts when designing the application. [1] https://www.powersync.com/ https://www.powersync.com/
- Sammi 2y ago
  My unfortunate point is that the dev cannot know what the user is doing, and so cannot in principle know what choice to make on behalf of the user in case of a conflict. This is not a code problem. It cannot be solved with code.
  - sgt 2y ago
    I've found that in almost all cases - the latest update "wins" strategy is fine. You could have two sessions working with conventional API calls and still have a conflict. As a dev you need to restrict what the user can do.
Jyaif 2y ago
> All this recent hype about sync engines and local first applications completely disregards conflict resolution The main concern of sync engines is precisely the conflict resolution! Everything else is simple in comparison. The good news is that under some circumstances it is possible to solve conflicts without user intervention. The simplest example is a counter that can only be incremented. More advanced data structures automatically solving conflicts exists, for example solving conflicts for strings exists, and those are good enough for a text editor. I agree that there will be conflicts that are resolved in a way that yields non-sensical text, for example if there are 2 edits of the sentence "One cat": One cat => Two cats One cat => One dog The resulting merge may be something like "Two cats dog". Something else (the user, an LLM...) will then have to fix it. But that's totally OK, because in practice this will happen extremely rarely, only when the user would have been offline for a long time. That user will be happy to have been able to work offline, largely compensating the fact that they have to proof read the text again.
- SkiFire13 2y ago
  This doesn't "solve" conflict resolution, it just picks one of the possible answers and then doesn't care whether it was the correct one or not. It can be acceptable for some usecases, but not for others where you're still concerned about stuff that happens "extremely rately" and is not under your direct control. > Something else (the user, an LLM...) will then have to fix it. This assumes that user/llm knows the conflict was automatically solved and might need to be fixed, so the conflict is still there! You just made the manual part delayed and non-mandatory, but if you want correctness it will still have to be there.
- brulard 2y ago
  > in practice this will happen extremely rarely, only when the user would have been offline for a long time. I don't think it would happen "extremely rarely". Drops in connectivity happen a lot, especially on cellular connection and this can absolutely happen a lot for some applications. Especially when talking about "offline first" apps.
  - Jyaif 2y ago
aboodman 2y ago
Zero (zerosync.dev) uses transactional conflict resolution, which is what our prior product Replicache and Reflect both used. It is very similar to what multiplayer games have done for decades. It is described here: https://rocicorp.dev/blog/ready-player-two https://rocicorp.dev/blog/ready-player-two It works really well and we and our customers have found it to be quite general. It allows you to run an arbitrary transaction on the sever side to decide what to do in case of conflicts. It is the software equivalent of git asking the user what to do. Zero asks your code what to do. But it asks it in the form of the question "please run the function named x with these inputs on the current backend db state". Which is a much more ergonomic way to ask it than "please do a 3-way merge between these three states". Conflict resolution is not the reason why there has not been a general-purpose sync engine. None of our customers have ~ever complained about conflict resolution. The reason there has not been a general-purpose sync engine is actually on the read side: - Previous sync engines really want you to sync all data. This is impractical for most apps. - Previous sync engines do not have practical approaches to permissions. These problems are being solved in next generation of sync engines. For more on this, I talk about it some here: https://www.youtube.com/watch?v=rqOUgqsWvbw https://www.youtube.com/watch?v=rqOUgqsWvbw
- probabletrain 2y ago
  I think with good presence (being able to see what other users are doing) and an app that isn't used offline, conflicts are essentially not a problem. As long as whatever is resolving the conflicts resolves them in a way that doesn't break the app, e.g. making sure there aren't cycles in some multiplayer app with a tree datastructure. Sounds like Zero has the right idea here, I'll build something on it imminently to try it out.
  - Sammi 2y ago
    Agree that if you don't have offline support, then conflict resolution is such a minor issue that you can just do "last write wins" and call it a day.
- probabletrain 2y ago
  > Previous sync engines really want you to sync all data Linear had to do all sorts of shenanigans to be able to sync all data, for orgs with lots of it – there's a talk on that here: https://www.youtube.com/watch?v=Wo2m3jaJixU&t=1473s https://www.youtube.com/watch?v=Wo2m3jaJixU&t=1473s
jamil7 2y ago
> All this recent hype about sync engines and local first applications Kind of but only really in the web world, it was the default on desktop for a long time and is pretty common on mobile.
deleted 2y ago
[deleted]

by Sammi 2y ago

porridgeraisin 2y ago

Precisely. The hype articles write all about the journey to The Wall, and then leave out the bit where you smash headfirst into it.

lifty 2y ago

Very good point. The local-sync ecosystem is still in a young phase, and conflict resolution hasn't been tackled or solved yet. Most systems have a |last write wins" approach.

sgt 2y ago

> All this recent hype about sync engines and local first applications completely disregards conflict resolution. Not really true though. I've used a couple of local sync engines, one internally built and another one which is both commercial and now open source called PowerSync[1]. Conflict resolution is definitely on the agenda, and a developer is definitely going to be mindful of conflicts when designing the application. [1] https://www.powersync.com/ https://www.powersync.com/

Sammi 2y ago
My unfortunate point is that the dev cannot know what the user is doing, and so cannot in principle know what choice to make on behalf of the user in case of a conflict. This is not a code problem. It cannot be solved with code.
- sgt 2y ago
  I've found that in almost all cases - the latest update "wins" strategy is fine. You could have two sessions working with conventional API calls and still have a conflict. As a dev you need to restrict what the user can do.

Jyaif 2y ago

> All this recent hype about sync engines and local first applications completely disregards conflict resolution The main concern of sync engines is precisely the conflict resolution! Everything else is simple in comparison. The good news is that under some circumstances it is possible to solve conflicts without user intervention. The simplest example is a counter that can only be incremented. More advanced data structures automatically solving conflicts exists, for example solving conflicts for strings exists, and those are good enough for a text editor. I agree that there will be conflicts that are resolved in a way that yields non-sensical text, for example if there are 2 edits of the sentence "One cat": One cat => Two cats One cat => One dog The resulting merge may be something like "Two cats dog". Something else (the user, an LLM...) will then have to fix it. But that's totally OK, because in practice this will happen extremely rarely, only when the user would have been offline for a long time. That user will be happy to have been able to work offline, largely compensating the fact that they have to proof read the text again.

SkiFire13 2y ago
This doesn't "solve" conflict resolution, it just picks one of the possible answers and then doesn't care whether it was the correct one or not. It can be acceptable for some usecases, but not for others where you're still concerned about stuff that happens "extremely rately" and is not under your direct control. > Something else (the user, an LLM...) will then have to fix it. This assumes that user/llm knows the conflict was automatically solved and might need to be fixed, so the conflict is still there! You just made the manual part delayed and non-mandatory, but if you want correctness it will still have to be there.
brulard 2y ago
> in practice this will happen extremely rarely, only when the user would have been offline for a long time. I don't think it would happen "extremely rarely". Drops in connectivity happen a lot, especially on cellular connection and this can absolutely happen a lot for some applications. Especially when talking about "offline first" apps.
- Jyaif 2y ago

aboodman 2y ago

Zero (zerosync.dev) uses transactional conflict resolution, which is what our prior product Replicache and Reflect both used. It is very similar to what multiplayer games have done for decades. It is described here: https://rocicorp.dev/blog/ready-player-two https://rocicorp.dev/blog/ready-player-two It works really well and we and our customers have found it to be quite general. It allows you to run an arbitrary transaction on the sever side to decide what to do in case of conflicts. It is the software equivalent of git asking the user what to do. Zero asks your code what to do. But it asks it in the form of the question "please run the function named x with these inputs on the current backend db state". Which is a much more ergonomic way to ask it than "please do a 3-way merge between these three states". Conflict resolution is not the reason why there has not been a general-purpose sync engine. None of our customers have ~ever complained about conflict resolution. The reason there has not been a general-purpose sync engine is actually on the read side: - Previous sync engines really want you to sync all data. This is impractical for most apps. - Previous sync engines do not have practical approaches to permissions. These problems are being solved in next generation of sync engines. For more on this, I talk about it some here: https://www.youtube.com/watch?v=rqOUgqsWvbw https://www.youtube.com/watch?v=rqOUgqsWvbw

probabletrain 2y ago
I think with good presence (being able to see what other users are doing) and an app that isn't used offline, conflicts are essentially not a problem. As long as whatever is resolving the conflicts resolves them in a way that doesn't break the app, e.g. making sure there aren't cycles in some multiplayer app with a tree datastructure. Sounds like Zero has the right idea here, I'll build something on it imminently to try it out.
- Sammi 2y ago
  Agree that if you don't have offline support, then conflict resolution is such a minor issue that you can just do "last write wins" and call it a day.
probabletrain 2y ago
> Previous sync engines really want you to sync all data Linear had to do all sorts of shenanigans to be able to sync all data, for orgs with lots of it – there's a talk on that here: https://www.youtube.com/watch?v=Wo2m3jaJixU&t=1473s https://www.youtube.com/watch?v=Wo2m3jaJixU&t=1473s

jamil7 2y ago

> All this recent hype about sync engines and local first applications Kind of but only really in the web world, it was the default on desktop for a long time and is pretty common on mobile.

deleted 2y ago

[deleted]