3 ms·
That's true in the general case, but there are usually ways to make it not matter. If you have a vector clock with snapshot reads, then you may not know consis
by GeneralMayhem 2y ago
That's true in the general case, but there are usually ways to make it not matter.
If you have a vector clock with snapshot reads, then you may not know consistently what the system looks like now, but you can know what it looked like as of some arbitrary previous point in time. If your use case doesn't require read-modify-writes (e.g., metric calculations where the source of truth is out of your control and you just need to aggregate deltas), then that's good enough - you can find a set of results that are in sync with each other, even if they're not fully up to date.
If you have limited read-modify-writes, but you can at least engineer synchronous transactions on subsets of the data (e.g., single-row-level transactions) with asynchronous replication and fan-out of side-effects, then you can turn more complex problems into something that looks like the first case.
Those tricks alone are enough to remove most of the weirdness with eventual-consistency, but the broader point is that distributed system design is more about learning a toolbox of tricks to handle and describe certain patterns, of which you may have any number in any combination in any given project, than it is about "solving" the problem once and for all.
- kukkeliskuu 2y agoSounds plausible. I am fighting a different battle, however. For my corporate customers, what you are describing is way too sophisticated. They do what others do. As Kafka is de facto messaging platform, everybody uses it. For example, if source data is multiplexed via Kafka into one sink, people typically say that the sink is "eventually consistent". Another case is when an operation writes into multiple microservices without transactions. People are not even consistent in their speech, let alone how they handle their data. It seems to me that in 99% of such corporate systems would work better if people would optimize integrity (i.e. communicate with real transactions on a real database) instead of optimizing for performance and scalability.