4 ms·
So I guess the specific issue that grandparent raised was if multiple users wanted to edit the same record. So even if the queue is sharded on some primary key,
by elvinyung 9y ago
So I guess the specific issue that grandparent raised was if multiple users wanted to edit the same record. So even if the queue is sharded on some primary key, then for each partition, either a) you only have one machine ingesting and thus a SPOF, or b) you ingest data using multiple machines, which means you need consensus/conflict resolution somewhere, right?
- Joeri 9y agoYou ingest the log partition in all machines to reconstruct the state from the log. You do this even on the machine that writes to the log, treating the log as the source of truth. Since every user would belong to one partition there would be a globally agreed ordering of their events in the log. Basically though that makes a CP tradeoff, since a network partition that hides the primary kafka replica from some of the writers causes writes to fail. There’s simply no way to have a globally distributed system that’s always available and always consistent, either you get inconsistency or you drop writes.
- elvinyung 9y agoYep, this was the answer I was expecting. Basically, I really doubt that event sourcing magically solves this, which is what the above comment claimed [1]. [1] https://news.ycombinator.com/item?id=16328127 https://news.ycombinator.com/item?id=16328127