3 ms·

Here's my take on Event Sourcing: it's not particularly well defined what an "event" is. Did the event happen yet? Did it succeed? Which part? I didn't like hav

Here's my take on Event Sourcing: it's not particularly well defined what an "event" is. Did the event happen yet? Did it succeed? Which part? I didn't like having an event ledger say "comment created," where my application is then meant to consume this, handle validation, potentially fail on the db operation, etc. So here is what I do: I basically combine Event Sourcing architecture with CQRS. Whenever a client makes a write request, that "command" gets written to a log. Then when things happen as a consequence of that command, for example, a successful DB write happens, I write another log that references the command. So I'll have an event that says: id: '1234' type: 'command' data: {type: 'createComment', text: 'Ayy', userId: '1234', postId: '6543'} And then that can get picked up and processed by the application, which will create another event like: id: '2345' type: 'dbWrite' data: {model: 'comments', data: {text: 'Ayy'}} commandId: '1234' There's a lot of redundancy, and you can instead rely on something like your DB WAL for some of the consequences of the commands, but separating out the commands from the results like this has made Event Sourcing work quite well for me.

ZenPsycho 8y ago
Having it clear not just in your own head, but your entire team's head what an event is, quite important, especially to keep in mind what happens when you "play back" a log. Do transactional emails get sent out again? Do upstream services record the playback as duplicate events? How are transactions handled? What external state are you unknowingly depending on for that playback to produce the same result?
- ralusek 8y ago
  Ya, I think that is part of the importance of separating out the effects that resulted from the command. If I want to benefit from the "state rewind" capabilities, to bring my DB back to a certain point in time, for example, I can simply play back the DB writes that occurred and perform those updates directly. Otherwise, I would have to replay the commands, ensure that my application state for processing the commands is exactly the same as it was when it last processed those particular commands, ensure that all undesirable side effects are disabled, etc.
- jacques_chester 8y ago
  > Do transactional emails get sent out again? Do upstream services record the playback as duplicate events? How are transactions handled? The narrow problem here is that Event Sourcing is temporal, but not bitemporal. Or rather, the different kinds of temporality are frequently muddled. Event streaming in the Akidau/Google style is bitemporal, but mostly accidentally, as a side effect of distinguishing "event time" (fact time) and "processing time" (transaction time / belief time). (Snodgrass later proposed tritemporal models, including a timeline for when a fact-belief was viewed, which I find both brilliant and slightly terrifying). The problems of evolving data models, which is the third-order problem, is often hardest when you put a log or stream at the centre of your design. Databases using SQL, for example, struggle with first-order (current state) and second-order (historical state) evolution. But they do much better on third-order evolution, since DDL is built into every relational database.
acjohnson55 8y ago
My separate comment was partially a reply to you: https://news.ycombinator.com/item?id=19073810 https://news.ycombinator.com/item?id=19073810
codebje 8y ago
> Did the event happen yet? Did it succeed? Events have already happened when you see them in the event log. It's happened, it succeeded. > I didn't like having an event ledger say "comment created," where my application is then meant to consume this, handle validation, potentially fail on the db operation, etc. You don't validate the past, you just accept it. Failing on the dB operation means your application is in error - if it's a transient error, retry with whatever backoff strategy you have. If it's a permanent error, you have a serious mistake somewhere in your code. In your example, I believe you've broken one event into two parts. The second part essentially only says "the command was accepted." The first part says why there was a change made. The first event is essentially "a user submitted a request to create a comment." The second event is essentially "a user's request resulted in a database change." The second event is not semantically interesting without the request's information, and the first event is of little interest outside of processing the request. Your application receives the _command_ to post a comment, validates the command, and records the outcome as an event. A complete system would record errors as events as well as successes. This might seem very noisy, but down the line you might need to answer questions like "was this user doing lots of failed comments leading up to their successful hack of our system?" Or "how many comments are failing because the spam filter rejected them?" And you can't answer those if you haven't recorded the error outcome. (And there's no harm in logging commands separately, too, but I'd log them explicitly as _commands_, not events. Commands are imperative tense, they're an order or request for the system to take some action. Events are past tense, they're what the system actually did.)

ZenPsycho 8y ago

Having it clear not just in your own head, but your entire team's head what an event is, quite important, especially to keep in mind what happens when you "play back" a log. Do transactional emails get sent out again? Do upstream services record the playback as duplicate events? How are transactions handled? What external state are you unknowingly depending on for that playback to produce the same result?

ralusek 8y ago
Ya, I think that is part of the importance of separating out the effects that resulted from the command. If I want to benefit from the "state rewind" capabilities, to bring my DB back to a certain point in time, for example, I can simply play back the DB writes that occurred and perform those updates directly. Otherwise, I would have to replay the commands, ensure that my application state for processing the commands is exactly the same as it was when it last processed those particular commands, ensure that all undesirable side effects are disabled, etc.
jacques_chester 8y ago
> Do transactional emails get sent out again? Do upstream services record the playback as duplicate events? How are transactions handled? The narrow problem here is that Event Sourcing is temporal, but not bitemporal. Or rather, the different kinds of temporality are frequently muddled. Event streaming in the Akidau/Google style is bitemporal, but mostly accidentally, as a side effect of distinguishing "event time" (fact time) and "processing time" (transaction time / belief time). (Snodgrass later proposed tritemporal models, including a timeline for when a fact-belief was viewed, which I find both brilliant and slightly terrifying). The problems of evolving data models, which is the third-order problem, is often hardest when you put a log or stream at the centre of your design. Databases using SQL, for example, struggle with first-order (current state) and second-order (historical state) evolution. But they do much better on third-order evolution, since DDL is built into every relational database.

acjohnson55 8y ago

My separate comment was partially a reply to you: https://news.ycombinator.com/item?id=19073810 https://news.ycombinator.com/item?id=19073810

codebje 8y ago

> Did the event happen yet? Did it succeed? Events have already happened when you see them in the event log. It's happened, it succeeded. > I didn't like having an event ledger say "comment created," where my application is then meant to consume this, handle validation, potentially fail on the db operation, etc. You don't validate the past, you just accept it. Failing on the dB operation means your application is in error - if it's a transient error, retry with whatever backoff strategy you have. If it's a permanent error, you have a serious mistake somewhere in your code. In your example, I believe you've broken one event into two parts. The second part essentially only says "the command was accepted." The first part says why there was a change made. The first event is essentially "a user submitted a request to create a comment." The second event is essentially "a user's request resulted in a database change." The second event is not semantically interesting without the request's information, and the first event is of little interest outside of processing the request. Your application receives the _command_ to post a comment, validates the command, and records the outcome as an event. A complete system would record errors as events as well as successes. This might seem very noisy, but down the line you might need to answer questions like "was this user doing lots of failed comments leading up to their successful hack of our system?" Or "how many comments are failing because the spam filter rejected them?" And you can't answer those if you haven't recorded the error outcome. (And there's no harm in logging commands separately, too, but I'd log them explicitly as _commands_, not events. Commands are imperative tense, they're an order or request for the system to take some action. Events are past tense, they're what the system actually did.)