6 ms·

I've seen CQRS pop up over and over, but never seen anyone do a real application aside from some pseudo-bank account code. Are there any bigger OSS projects tha

I've seen CQRS pop up over and over, but never seen anyone do a real application aside from some pseudo-bank account code. Are there any bigger OSS projects that use event sourcing? Also the cached version: https://webcache.googleusercontent.com/search?q=cache:jZXfTYYOUnUJ:https://altkomsoftware.pl/en/blog/cqrs-event-sourcing/+&cd=1&hl=de&ct=clnk&gl=de&client=ubuntu https://webcache.googleusercontent.com/search?q=cache:jZXfTY...

mrkeen 7y ago
Used it. Loved it. It does have its issues though. I was working on the backend systems for electric car charging. When we heard about real-world events happen (start-charging, stop-charging, etc.), we wrote them directly to Kafka. It was up to other services to interpret those events, e.g. "I saw a start then a stop, so I'm writing an event to say user-has-debt". Yet another service says "I see a debt, I'm going to try to fix this by charging a credit card". I guess you'd call the above the 'C' part of CQRS. But Kafka by itself is not great for relational queries. So we had additional services for, e.g history. The history service also listened to starts, stops, debts, credits, etc. and built up a more traditional SQL table optimised for relational queries, so a user could quickly see where they had charged before. The issues we had were: 1) Where's the REST/Kafka boundary? I.e. when should something write to the Kafka log as opposed to POSTing directly to another service? E.g. If a user sets their payment method, do we update some DB immediatley, or do we write the fact that they set their payment method onto Kafka, and have another service read it? 2) Services which had to replay from the beginning of time took a while to start up, so we had to find ways to get them not to. 3) You need to be serious about versioning. Since many services read messages from Kafka, you can't just change the implementation of those messages. We explicitly versioned every event in its class name. Worth it? For our use case, hell yeah.
- dmitryminkovsky 7y ago
  > 1) Where's the REST/Kafka boundary? I.e. when should something write to the Kafka log as opposed to POSTing directly to another service? E.g. If a user sets their payment method, do we update some DB immediatley, or do we write the fact that they set their payment method onto Kafka, and have another service read it? I believe the term that's emerging for this issue is "collapsing CQRS" and how you handle this is application-dependent (are you using plain synchronous HTTP requests? websockets?) In my case, the HTTP server has a producer that writes to a Kafka topic and a consumer that consumes the answers. The HTTP request waits until the answer appears on the answer topic. > 2) Services which had to replay from the beginning of time took a while to start up, so we had to find ways to get them not to. Kafka Streams local state makes this fast, unless you need to reprocess. > 3) You need to be serious about versioning. Since many services read messages from Kafka, you can't just change the implementation of those messages. We explicitly versioned every event in its class name. Yes, this is tricky. In my case, I either add fields in a backwards compatible manner, or rebase/rewrite event topics and roll them out while unwinding the previous version of the topic that may still be in use. The former is obviously the simpler option.
namelosw 7y ago
I'm not totally sure but Git might be one?
Heliosmaster 7y ago
We've implemented it in a E-learning system used in some Dutch High-schools. A few talks we've given at the local Clojure Meetup: - https://zeekat.nl/news/2015/02/12/event-sourcing-at-studyflow-slides/es-at-sf.pdf https://zeekat.nl/news/2015/02/12/event-sourcing-at-studyflo... - https://speakerdeck.com/helios/2 https://speakerdeck.com/helios/2 - https://speakerdeck.com/helios/event-sourcing-cqrs-and-scalability-a-status-update https://speakerdeck.com/helios/event-sourcing-cqrs-and-scala... Right now the system is running with ~800 million events, with roughly 1.5M added per day :)
agentultra 7y ago
Most relational database engines use some form of event-sourcing internally.
- corn_dog 7y ago
  Makes you wonder why people want to reinvent everything from scratch then.
dmitryminkovsky 7y ago
My current project is totally ES/CRQS from the core to the UI. It's an email service with users, messages, client sessions and other entities. Please see my top-level comment about the joy of implementing it with Kafka Streams. These state of the art is still emerging, so it was a lot of work to figure out what tools to use and how to model things, but once I did, I can't look back update state in a table again.

mrkeen 7y ago

Used it. Loved it. It does have its issues though. I was working on the backend systems for electric car charging. When we heard about real-world events happen (start-charging, stop-charging, etc.), we wrote them directly to Kafka. It was up to other services to interpret those events, e.g. "I saw a start then a stop, so I'm writing an event to say user-has-debt". Yet another service says "I see a debt, I'm going to try to fix this by charging a credit card". I guess you'd call the above the 'C' part of CQRS. But Kafka by itself is not great for relational queries. So we had additional services for, e.g history. The history service also listened to starts, stops, debts, credits, etc. and built up a more traditional SQL table optimised for relational queries, so a user could quickly see where they had charged before. The issues we had were: 1) Where's the REST/Kafka boundary? I.e. when should something write to the Kafka log as opposed to POSTing directly to another service? E.g. If a user sets their payment method, do we update some DB immediatley, or do we write the fact that they set their payment method onto Kafka, and have another service read it? 2) Services which had to replay from the beginning of time took a while to start up, so we had to find ways to get them not to. 3) You need to be serious about versioning. Since many services read messages from Kafka, you can't just change the implementation of those messages. We explicitly versioned every event in its class name. Worth it? For our use case, hell yeah.

dmitryminkovsky 7y ago
> 1) Where's the REST/Kafka boundary? I.e. when should something write to the Kafka log as opposed to POSTing directly to another service? E.g. If a user sets their payment method, do we update some DB immediatley, or do we write the fact that they set their payment method onto Kafka, and have another service read it? I believe the term that's emerging for this issue is "collapsing CQRS" and how you handle this is application-dependent (are you using plain synchronous HTTP requests? websockets?) In my case, the HTTP server has a producer that writes to a Kafka topic and a consumer that consumes the answers. The HTTP request waits until the answer appears on the answer topic. > 2) Services which had to replay from the beginning of time took a while to start up, so we had to find ways to get them not to. Kafka Streams local state makes this fast, unless you need to reprocess. > 3) You need to be serious about versioning. Since many services read messages from Kafka, you can't just change the implementation of those messages. We explicitly versioned every event in its class name. Yes, this is tricky. In my case, I either add fields in a backwards compatible manner, or rebase/rewrite event topics and roll them out while unwinding the previous version of the topic that may still be in use. The former is obviously the simpler option.

namelosw 7y ago

I'm not totally sure but Git might be one?

Heliosmaster 7y ago

We've implemented it in a E-learning system used in some Dutch High-schools. A few talks we've given at the local Clojure Meetup: - https://zeekat.nl/news/2015/02/12/event-sourcing-at-studyflow-slides/es-at-sf.pdf https://zeekat.nl/news/2015/02/12/event-sourcing-at-studyflo... - https://speakerdeck.com/helios/2 https://speakerdeck.com/helios/2 - https://speakerdeck.com/helios/event-sourcing-cqrs-and-scalability-a-status-update https://speakerdeck.com/helios/event-sourcing-cqrs-and-scala... Right now the system is running with ~800 million events, with roughly 1.5M added per day :)

agentultra 7y ago

Most relational database engines use some form of event-sourcing internally.

corn_dog 7y ago
Makes you wonder why people want to reinvent everything from scratch then.

dmitryminkovsky 7y ago

My current project is totally ES/CRQS from the core to the UI. It's an email service with users, messages, client sessions and other entities. Please see my top-level comment about the joy of implementing it with Kafka Streams. These state of the art is still emerging, so it was a lot of work to figure out what tools to use and how to model things, but once I did, I can't look back update state in a table again.