4 ms·

One of the problems I think I see with event sourcing is its inability to scale. You have to guarantee the order of events, right? How do you do that in a large

One of the problems I think I see with event sourcing is its inability to scale. You have to guarantee the order of events, right? How do you do that in a large scale distributed system with eventual consistency, without incurring an insane synchronization time penalty? I'd genuinely love to hear if you have a good solution for this, because if you do I have a use case I need it for, so this ain't a troll comment!

jdc 8y ago
https://stackoverflow.com/questions/41082938/event-sourcing-microservices-how-to-manage-timestamp https://stackoverflow.com/questions/41082938/event-sourcing-...
- ta93754829 8y ago
  that basically says Vector Clock, which is a good solution for where it's usually applied, but I think this is different. Having everything bottlenecks through your vector clock isn't going to be feasible.
latencyloser 8y ago
You generally only have to guarantee the ordering of _some_ events--not all of them. Ideally you want a way to partition your data such that ordering is only important per partition. For example, when I worked in derivatives trading, we only needed to guarantee ordering by product class, but events could be generally unordered given multiple product classes, as long as each individual one was ordered. Similarly, the telemetry systems I work on now guarantee ordering per sensor (duh) and per sensor grouping, but not across _all_ sensors on the vehicle. Partitioning makes for easy scaling (just make a new partition yay).
ZenPsycho 8y ago
I guess, whatever a Kafka cluster does, that's how you do it.
ludston 8y ago
This depends on the nature of your scaling issue. Are you having trouble with consuming events? Or with adding all of your events whilst trying to ensure that they are ordered? For the former, you might use something like [ZooKeeper](https://www.elastic.co/blog/found-zookeeper-king-of-coordination https://www.elastic.co/blog/found-zookeeper-king-of-coordina...) in your consumers to control parallelisation. For the latter, you order by time of insertion. I think you might have some contraints you haven't mentioned that may be the real problem you are trying to solve. What is distributed?
romaneremin 8y ago
Exact order of events should be guaranteed for one DDD aggregate, it is doable with simple optimistic locking. So unless your system consists of one aggregate - you should not have scalability issues.

jdc 8y ago

https://stackoverflow.com/questions/41082938/event-sourcing-microservices-how-to-manage-timestamp https://stackoverflow.com/questions/41082938/event-sourcing-...

ta93754829 8y ago
that basically says Vector Clock, which is a good solution for where it's usually applied, but I think this is different. Having everything bottlenecks through your vector clock isn't going to be feasible.

latencyloser 8y ago

You generally only have to guarantee the ordering of _some_ events--not all of them. Ideally you want a way to partition your data such that ordering is only important per partition. For example, when I worked in derivatives trading, we only needed to guarantee ordering by product class, but events could be generally unordered given multiple product classes, as long as each individual one was ordered. Similarly, the telemetry systems I work on now guarantee ordering per sensor (duh) and per sensor grouping, but not across _all_ sensors on the vehicle. Partitioning makes for easy scaling (just make a new partition yay).

ZenPsycho 8y ago

I guess, whatever a Kafka cluster does, that's how you do it.

ludston 8y ago

This depends on the nature of your scaling issue. Are you having trouble with consuming events? Or with adding all of your events whilst trying to ensure that they are ordered? For the former, you might use something like [ZooKeeper](https://www.elastic.co/blog/found-zookeeper-king-of-coordination https://www.elastic.co/blog/found-zookeeper-king-of-coordina...) in your consumers to control parallelisation. For the latter, you order by time of insertion. I think you might have some contraints you haven't mentioned that may be the real problem you are trying to solve. What is distributed?

romaneremin 8y ago

Exact order of events should be guaranteed for one DDD aggregate, it is doable with simple optimistic locking. So unless your system consists of one aggregate - you should not have scalability issues.