4 ms·

I think the issue is solved (while still complex) with an event sourcing system. The chances are really high that the users did not push / updated the number at

by merb 9y ago

I think the issue is solved (while still complex) with an event sourcing system. The chances are really high that the users did not push / updated the number at the exact same time. (so they are at least 1ns apart, even when not it's not that bad). So you can actually restore to a sane state by reapplying the log from scratch. i.e. this is solved with CQRS and Event Sourcing and it probably works in like all databases. It's quite complex but pretty reliable, I'm pretty sure that everybody already built at least a extremly simple append only event log.

elvinyung 9y ago
Does this imply that there's a single place that log entries get ingested at? Doesn't that make it a single point of failure?
- merb 9y ago
  well you can either use a single database for log entries, or cassandra or a mysql cluster or a pg cluster. that doesn't matter.. basially instead of having a table where you update/delete entries you basically append only to a table. so insert only. you can still have another table that will be aggregated from your log table. where you store (i.e. which database system, cluster whatever) your stuff doesn't matter.
  - elvinyung 9y ago
    That doesn't answer my question. If you're appending to the end of a table, regardless of system you still need one machine somewhere to ingest new records for the same primary key, right?
- cube2222 9y ago
  No, you can just have partitioned replicated topics like kafka has, just make sure that all the events for one user get put into the same partition, so that they get totally ordered.
  - elvinyung 9y ago
    So I guess the specific issue that grandparent raised was if multiple users wanted to edit the same record. So even if the queue is sharded on some primary key, then for each partition, either a) you only have one machine ingesting and thus a SPOF, or b) you ingest data using multiple machines, which means you need consensus/conflict resolution somewhere, right?
    - Joeri 9y ago
      You ingest the log partition in all machines to reconstruct the state from the log. You do this even on the machine that writes to the log, treating the log as the source of truth. Since every user would belong to one partition there would be a globally agreed ordering of their events in the log. Basically though that makes a CP tradeoff, since a network partition that hides the primary kafka replica from some of the writers causes writes to fail. There’s simply no way to have a globally distributed system that’s always available and always consistent, either you get inconsistency or you drop writes.
      - elvinyung 9y ago
        Yep, this was the answer I was expecting. Basically, I really doubt that event sourcing magically solves this, which is what the above comment claimed [1]. [1] https://news.ycombinator.com/item?id=16328127 https://news.ycombinator.com/item?id=16328127
lukev 9y ago
1ns according to whom? Clock skew is still a thing :)

by merb 9y ago

elvinyung 9y ago

Does this imply that there's a single place that log entries get ingested at? Doesn't that make it a single point of failure?

merb 9y ago
well you can either use a single database for log entries, or cassandra or a mysql cluster or a pg cluster. that doesn't matter.. basially instead of having a table where you update/delete entries you basically append only to a table. so insert only. you can still have another table that will be aggregated from your log table. where you store (i.e. which database system, cluster whatever) your stuff doesn't matter.
- elvinyung 9y ago
  That doesn't answer my question. If you're appending to the end of a table, regardless of system you still need one machine somewhere to ingest new records for the same primary key, right?
cube2222 9y ago
No, you can just have partitioned replicated topics like kafka has, just make sure that all the events for one user get put into the same partition, so that they get totally ordered.
- elvinyung 9y ago
  So I guess the specific issue that grandparent raised was if multiple users wanted to edit the same record. So even if the queue is sharded on some primary key, then for each partition, either a) you only have one machine ingesting and thus a SPOF, or b) you ingest data using multiple machines, which means you need consensus/conflict resolution somewhere, right?
  - Joeri 9y ago
    You ingest the log partition in all machines to reconstruct the state from the log. You do this even on the machine that writes to the log, treating the log as the source of truth. Since every user would belong to one partition there would be a globally agreed ordering of their events in the log. Basically though that makes a CP tradeoff, since a network partition that hides the primary kafka replica from some of the writers causes writes to fail. There’s simply no way to have a globally distributed system that’s always available and always consistent, either you get inconsistency or you drop writes.
    - elvinyung 9y ago
      Yep, this was the answer I was expecting. Basically, I really doubt that event sourcing magically solves this, which is what the above comment claimed [1]. [1] https://news.ycombinator.com/item?id=16328127 https://news.ycombinator.com/item?id=16328127

lukev 9y ago

1ns according to whom? Clock skew is still a thing :)