3 ms·
> Lastly, the approach I never explored but was curious about is the idea of the client being responsible for pushing all values to all nodes rather than replic
by hodgesrm 3y ago
> Lastly, the approach I never explored but was curious about is the idea of the client being responsible for pushing all values to all nodes rather than replication happening in the background.
You can client replication at industrial scale by having clients push writes to Kafka and then have multiple, independent systems read and apply them. You still have a SPOF on Kafka of course. Another practical issue is that if you take this approach you'll likely need a utility to detect and correct data drift. That's been a feature of the systems I've seen that use this approach successfully.
- IgorPartola 3y agoSo what I was picturing was more of where the client directly pushes to all the backend nodes simultaneously. That way the client right away knows if it was a success or a failure or a partial success. The whole problem with any of this is that you can have writes be a partial success and what that means cannot be correctly interpreted by the database cluster alone in the general case. Like it’s bad when it happens but is catastrophic or can it be recovered from really depends on what the data is and it should be up to the application to decide that. Instead we are trying to treat databases like it’s always success or failure with maaaaybe some eventual consistency semantics and when a partition happens or a node is simply down the application doesn’t really know if any given piece of data is actually fully durably committed or just partially committed.