4 ms·

Does this imply that there's a single place that log entries get ingested at? Doesn't that make it a single point of failure?

merb 9y ago
well you can either use a single database for log entries, or cassandra or a mysql cluster or a pg cluster. that doesn't matter.. basially instead of having a table where you update/delete entries you basically append only to a table. so insert only. you can still have another table that will be aggregated from your log table. where you store (i.e. which database system, cluster whatever) your stuff doesn't matter.
- elvinyung 9y ago
  That doesn't answer my question. If you're appending to the end of a table, regardless of system you still need one machine somewhere to ingest new records for the same primary key, right?
cube2222 9y ago
No, you can just have partitioned replicated topics like kafka has, just make sure that all the events for one user get put into the same partition, so that they get totally ordered.
- elvinyung 9y ago
  So I guess the specific issue that grandparent raised was if multiple users wanted to edit the same record. So even if the queue is sharded on some primary key, then for each partition, either a) you only have one machine ingesting and thus a SPOF, or b) you ingest data using multiple machines, which means you need consensus/conflict resolution somewhere, right?
  - Joeri 9y ago
    You ingest the log partition in all machines to reconstruct the state from the log. You do this even on the machine that writes to the log, treating the log as the source of truth. Since every user would belong to one partition there would be a globally agreed ordering of their events in the log. Basically though that makes a CP tradeoff, since a network partition that hides the primary kafka replica from some of the writers causes writes to fail. There’s simply no way to have a globally distributed system that’s always available and always consistent, either you get inconsistency or you drop writes.
    - elvinyung 9y ago
      Yep, this was the answer I was expecting. Basically, I really doubt that event sourcing magically solves this, which is what the above comment claimed [1]. [1] https://news.ycombinator.com/item?id=16328127 https://news.ycombinator.com/item?id=16328127

merb 9y ago

well you can either use a single database for log entries, or cassandra or a mysql cluster or a pg cluster. that doesn't matter.. basially instead of having a table where you update/delete entries you basically append only to a table. so insert only. you can still have another table that will be aggregated from your log table. where you store (i.e. which database system, cluster whatever) your stuff doesn't matter.

elvinyung 9y ago
That doesn't answer my question. If you're appending to the end of a table, regardless of system you still need one machine somewhere to ingest new records for the same primary key, right?

cube2222 9y ago

No, you can just have partitioned replicated topics like kafka has, just make sure that all the events for one user get put into the same partition, so that they get totally ordered.

elvinyung 9y ago
So I guess the specific issue that grandparent raised was if multiple users wanted to edit the same record. So even if the queue is sharded on some primary key, then for each partition, either a) you only have one machine ingesting and thus a SPOF, or b) you ingest data using multiple machines, which means you need consensus/conflict resolution somewhere, right?
- Joeri 9y ago
  You ingest the log partition in all machines to reconstruct the state from the log. You do this even on the machine that writes to the log, treating the log as the source of truth. Since every user would belong to one partition there would be a globally agreed ordering of their events in the log. Basically though that makes a CP tradeoff, since a network partition that hides the primary kafka replica from some of the writers causes writes to fail. There’s simply no way to have a globally distributed system that’s always available and always consistent, either you get inconsistency or you drop writes.
  - elvinyung 9y ago
    Yep, this was the answer I was expecting. Basically, I really doubt that event sourcing magically solves this, which is what the above comment claimed [1]. [1] https://news.ycombinator.com/item?id=16328127 https://news.ycombinator.com/item?id=16328127