3 ms·
These days quite a lot of our problems are synchronisation problems (are you pulling mutable data regularly from a remote API and storing it? thats actually a s
by codeulike 3y ago
These days quite a lot of our problems are synchronisation problems (are you pulling mutable data regularly from a remote API and storing it? thats actually a syncronisation) but despite that synchronisation and how complicated it can be doesnt seem to get discussed. You don't hear debates about different approaches, we dont seem to have a taxonomy of different techniques.
People seem to forget how hard it is. So lets say you're pushing changes from one system to another. How are you detecting changes? timestamps or a direct diff? Or checksums? Have you remembered to think about deletions? Can you even detect deletions? How do you keep track of which bits have successfully synced and which havent? If something unusual goes wrong (network goes down) halfway through a sync can you recover from that? Incremental sync inevitably drifts over time due to bugs and downtime, so are you also building a full-scale wipe-and-refresh or compare-and-reconcile to accompany the incremental sync? And dont even get me started on two-way syncs and conflict resolution and how hard that is to do without some sort of review-by-human component.
- dathinab 3y agoI think the reason why it's not discussed that much is because a lot of software gave up on properly synchronizing changes and go with a "handwaving eventual consistency" approach for a lot of things. Furthermore decisions about which db is used are rarely driven by the synchronization model. Except maybe sometimes the question "eventual consistency yes/no/a bit only". And in academia (wrt. common use-cases) we do have a bunch of well analyzed ways to archive synchronization the main question often isn't one of academic research needing to be done but of which model with which parameters and implementation details on top to use for a specific kind of db. Which mainly is discussed by devs when writing new dbs, i.e. not a everyday thing. While for the many new sqlite sync approaches the answers is nearly always "handwaving eventual consistency", cause nothing else would work on the edge anyway.
- jgaa 3y ago> "handwaving eventual consistency" approach for a lot of things. Which, for some systems translates to: It will be in sync when all the entropy in the universe is gone.
- dathinab 3y agoit's also a bit of an red hearing argument sure some systems under correct operation will never ever be fully in sync as long as they are used but most systems also do not need to be ever fully in sync it's good enough that for a specific context they will be in sync in not to much time if that context stops changing and that is something they do provide e.g. after updating a JSON blob stored under a specific id that update will be eventually available in the not too distant future and if no future changes to that document happen then in the context of that document the system will be fully in sync. But because you have very man documents there will always be a document which isn't yet in sync and in turn the system as a whole will never be fully in sync. But in the end that doesn't really matter. Also: no computer system will live that long, and if they aren't put in sync before they stop working they will literally never be fully in sync.
- Piskvorrr 3y agoTLDR: make it a Someone Else's Problem, in other words "not synced? Our code works as designed, go bother Support, or whatever."
- dathinab 3y agono its not somones else problem it's still your system but a system as a whole never being fully in sync is a red herring argument which misses the point and focuses on an aspect which might sound like a problem but hardly ever is a problem at all