4 ms·

I've had several clients lately who have either built or needed to rescue an architecture that worked from events in a log. I'm amazed at how many teams seem to

I've had several clients lately who have either built or needed to rescue an architecture that worked from events in a log. I'm amazed at how many teams seem to get it wrong. Like the author here, there are apparently a lot of different ways to interpret the design, and therefore a lot of subtle mistakes to make. I tried to summarize what I've learned in "Why are software developers confused by Kafka and immutable logs?" http://www.smashcompany.com/technology/why-are-software-developers-confused-by-kafka-and-immutable-logs http://www.smashcompany.com/technology/why-are-software-deve...

pondidum 7y ago
I agree with your post but...You almost certainly shouldn't be doing EventSourcing with Kafka. EventStreaming sure, Kafka is pretty idea for this (as your source of truth). EventSourced aggregates need to reload their entire history (unless you are snapshotting too) before applying a command, so you shouldn't have all your aggregates in a single Kafka topic, as you would have to read all events since forever, for all aggregates, every time you loaded one. If you put each aggregate into a separate topic, everything is fine, until you want to add projections - how do your projections know to subscribe to all the new topics being created for each aggregate? You can probably engineer your way around these limitations, but at that point, why not use a tool which is more suitable for the eventsourcing and keep Kafka for the "Business Process Event"?
- lkrubner 7y ago
  About this: “how do your projections know to subscribe to all the new topics being created for each aggregate?” Isn’t that precisely why people use ZooKeeper?
arkh 7y ago
It is because of the usual examples taken to show off ES. Wow! I can have a todo-list with easy undo-redo-branch! With free audit logs. We need audit logs for our new project. Forget how your domain application must handle permissions, forget about calling third-party API or sending mails. Forget about having to remove some of the data you have about a user when they ask for it. Anyway that's the maintenance team problem, you got your bonus and are now on a new project to try those "microservice things" to pad your resume.

pondidum 7y ago

I agree with your post but...You almost certainly shouldn't be doing EventSourcing with Kafka. EventStreaming sure, Kafka is pretty idea for this (as your source of truth). EventSourced aggregates need to reload their entire history (unless you are snapshotting too) before applying a command, so you shouldn't have all your aggregates in a single Kafka topic, as you would have to read all events since forever, for all aggregates, every time you loaded one. If you put each aggregate into a separate topic, everything is fine, until you want to add projections - how do your projections know to subscribe to all the new topics being created for each aggregate? You can probably engineer your way around these limitations, but at that point, why not use a tool which is more suitable for the eventsourcing and keep Kafka for the "Business Process Event"?

lkrubner 7y ago
About this: “how do your projections know to subscribe to all the new topics being created for each aggregate?” Isn’t that precisely why people use ZooKeeper?

arkh 7y ago

It is because of the usual examples taken to show off ES. Wow! I can have a todo-list with easy undo-redo-branch! With free audit logs. We need audit logs for our new project. Forget how your domain application must handle permissions, forget about calling third-party API or sending mails. Forget about having to remove some of the data you have about a user when they ask for it. Anyway that's the maintenance team problem, you got your bonus and are now on a new project to try those "microservice things" to pad your resume.