11 ms·

Event Sourcing (2005)

105 points by gjvc 5y ago

swyx 5y ago
We debated the precise meaning of Event Sourcing when I was writing my Temporal.io explainer (https://www.swyx.io/why-temporal/ https://www.swyx.io/why-temporal/) Basically the source of the debate - to do "proper" event sourcing, do you need to rerun the computations each time you roll back/forward, or is it enough to simply restore state?
- bluepizza 5y ago
  Hello. I love Temporal, it is an amazing solution. I have been pushing it pretty strongly at my workplace, but you guys need to make that web interface better. I lose way too many people at the demo stage. Next year on the Q1 I'll be able to push for a full demo again. Hope we get traction this time. It's great tech, and a pleasure to use.
  - swyx 5y ago
    i'm literally working on it as we speak :) hear you loud and clear. my top 2 priorities rn are TypeScript SDK and new Web UI.
- dtornow 5y ago
  I was part of that debate, I remember a rather interesting point of discussion: Is the main operation "apply" or "dedup" Apply seems to be the common notion of event sourcing: There is a function apply that takes a state an event and yields a new state. Then, starting in an init state and iteratively applying the entire event history, boom, latest state restored. Dedup has a lot of charm though: Run and rerun your code, if that step of your code is executed for the first time (no corresponding event in the event history) execute the step and store its result as an event in the history, however, if that step of your code is executed for the second, third time (there is a corresponding event in the event history) do not execute the step and return its result from the event in the history. The Haskell Workflow Package (https://hackage.haskell.org/package/Workflow https://hackage.haskell.org/package/Workflow) is a good example Temporal follows the second approach, so "proper" Event Sourcing? You be the judge :)
  - btown 5y ago
    This may be how Temporal works, but I imagine the way you ask "is there a corresponding event in the event history" is that you tag every "derived event" in the system with a hash of both the state and code used to create it? So if you change the code, you (eventually?) invalidate all events derived from it? This of course would be incredibly reliant on the code not being able to access anything except the state, and have no side effects - so if you want to say "if X, fetch Y" you need to have something looking at derived state from the "if X" part and putting the results of "fetch Y" into the event stream, thus causing the state relevant to the code to change, and invalidating the need for something to "fetch Y" a second time? There's something incredibly clean and enticing about this, but also incredibly scary to implement correctly at scale!
    - mfateev 5y ago
      Temporal takes much simpler approach to updating code while a program is running. For any code update, it keeps both old and the new version of the code. Then it uses the old code to replay already recorded events which allows to reconstruct the program state and takes the new code path when it is executed for the first time. Eventually the old code is removed once all the executions that used it are completed.
- acjohnson55 5y ago
  Can't remember how I first encountered your online presence, but I'm pretty sure I've followed you on Twitter for a while. Didn't realize you had joined Temporal. I've followed Temporal for a while, although I haven't yet had the chance to use it professionally. Still, I'm fairly convinced it's The Future, especially as it becomes obvious how to combine it with UX and data layer (right now, it's a bit unclear to me how much of UX state and permanent storage should live in Temporal). If I were building a green field business with automation at its core, I'd very strongly consider picking Temporal.
  - swyx 5y ago
    thank you! pls let us know if we can help with those questions when its time to evaluate us seriously.
- pyrale 5y ago
  That's a technical detail imo, as long as your implementation guarantees the outcomes are identical. The reason some people replay events is that it's a simple and easy way to restore previous state.
bob1029 5y ago
This is the direction I want to take our product. Running RCA on one of our transactions is really difficult because we only ever store the current state of affairs. I am still struggling with the grain on which I would want to source events. The engineer in me says its safest to go low-level and source key-value, so it doesnt matter if the biz model changes. But, this would reduce fidelity on the other side in terms of scanning for specific events of importance (e.g. SetKeyValue vs SetCustomerEmail).
- dharmaturtle 5y ago
  I would argue that there is no such thing as "the business model". There may be a projected rolled up summary state... but that's only one possible view out of infinitely many. For example, one possible "business model" could be "customers who have changed their emails in the past month". You shouldn't favor any particular view as being the favored "business model"... unless you're doing snapshots, in which case everything goes out the window. Basically, I vote for `SetCustomerEmail`.
  - Thiez 5y ago
    Events have already happened. The idiomatic name would be `CustomerEmailSet` or perhaps just `EmailSet` or `EmailChanged`.
- timClicks 5y ago
  Start with very granular events. They can be aggregated later if needed, but the reverse isn't possible.
  - rphln 5y ago
    Isn't it the other way around? If you have an “AB” event, it's fairly trivial to go through the log and split it into “A then B”. If you have “A then C then B” for whatever reason and then realize that you need to aggregate “A” and “B”, that is much harder.
- loldot_ 5y ago
erikerikson 5y ago
(2005)
- dang 5y ago
  Added. Thanks!
charles_f 5y ago
Posted 5 times but never really discussed https://hn.algolia.com/?q=https%3A%2F%2Fmartinfowler.com%2FeaaDev%2FEventSourcing.html https://hn.algolia.com/?q=https%3A%2F%2Fmartinfowler.com%2Fe...
- gjvc 5y ago
  Was this time though...
criticaltinker 5y ago
> Reversal is the most straightforward when the event is cast in the form of a difference. An example of this would be "add $10 to Martin's account" as opposed to "set Martin's account to $110". In the former case I can reverse by just subtracting $10, but in the latter case I don't have enough information to recreate the past value of the account. I've used the event sourcing pattern with great success in the context of an exceptionally complex web application for my day job. The front-end would create an 'event' for every action the user took. This allowed us to seamlessly implement client-side undo/redo functionality with surprisingly little effort, a market leading feature our users really appreciated. One key insight we had was the necessity of capturing the state change as a difference, as Martin explains. This turned out to be tremendously powerful. We were able to build a variety of other features on top including detailed analytics of how users were using our app, and session replay for training and debugging, among other things.
- sbazerque 5y ago
  from 10000ft away that has the feeling of op-based CRDTs
  - fennecfoxen 5y ago
    More or less.
  - swagasaurus-rex 5y ago
    Event sourcing can be represented as a chronological list of changes, replayed so that any list index represents a state. How CRDTs work is different in different implementations. Some CRDTs use a timestamp to order all changes, thus making it essentially identical to an event sourced list. Timestamp based ordering may not actually be the most ideal, especially for low trust environments where the timestamp can be spoofed. Some apps might not last-write wins. In this case, timestamp largely becomes meaningless because there has to be some other way to resolve conflicts due to multiple clients making changes to the same piece of data. Some apps want one clear winner, other apps might want the two inputs to be merged, others (like git) explicitly ask the developer to resolve the conflict. In this way CRDTs can diverge significantly from an ordered list of changes.
dharmaturtle 5y ago
Event sourcing is really good for building occasionally connected applications (e.g. phone app). Makes conflict resolution so much easier than, say two relational databases. Basically it follows the git model - distributed systems are each their own sources of truth, and allows for (relatively) simple syncing of state. When two distributed systems conflict, well... you need to write some kind of merge algorithm that may or may not ask for user input. But at least the accidental complexity is significantly reduced.
dminor 5y ago
I often find myself wishing for some sort of small scale durable log, so that this pattern was easy to implement in a small app backend without standing up something like Kafka.
- emerongi 5y ago
  I recently refactored a small set of services to use events instead of depending directly on eachother. I initially had Kafka in mind, but it would've been an absolute overkill. A "simple Kafka" would definitely be nice for these cases. I ended up using Redis Streams [0], which was good enough for my small-scale need. We already had Redis in our stack too, so it was a very simple integration. [0] https://redis.io/topics/streams-intro https://redis.io/topics/streams-intro
- uDontKnowMe 5y ago
  Have you considered rabbitmq or nats?
- dharmaturtle 5y ago
  What's wrong with EventStore? https://www.eventstore.com/ https://www.eventstore.com/
  - dminor 5y ago
    Nice! Hadn't heard of it, will check it out.
    - dharmaturtle 5y ago
      They have a channel in this slack that's pretty good for getting answers to questions: https://github.com/ddd-cqrs-es/slack-community https://github.com/ddd-cqrs-es/slack-community
- dimitrios1 5y ago
  try jetstream
- 5y ago
gjvc 5y ago
TDD stole much, if not all, of the airtime Event Sourcing should have received.
vmurthy 5y ago
As a techie-turned-PM , I found this pattern really useful when discussing specs with different stakeholders. Business stakeholders understand their business as a series of events (Customer walked in, customer bought X , customer checked out etc) so the user stories become a lot more relatable to both developers and business folks. I vaguely remember events being discussed in "Domain modelling made functional" , too
- gjvc 5y ago
  Very much agree. I had a memorable conversation with a PM a short time before the pandemic about this very subject. She was enthused about how it would make her work more closely aligned that of developers.
swagasaurus-rex 5y ago
I recently built https://github.com/siriusastrebe/jsynchronous https://github.com/siriusastrebe/jsynchronous which uses event sourcing to synchronize javascript variables on the server with connected browsers. You can also replay states using a special “rewind” mode, a core advantage of event sourcing.
acjohnson55 5y ago
Event sourcing is a game changer. Everywhere I have worked has ended up reimplementing it in some form, because it is irresistible to incorporate history into your data model. My most thorough use of event sourcing was in a real-time art auction system, which combined in-person and online bidding. An operator had to input the actions in the room into the system and bid on behalf of online bidders. Event sourcing allowed us to model what actually happened for each auction lot. It also made it possible to model undos and cancellations in really slick ways. Command-query response segregation (CQRS) is also a powerful concept, and it combines elegantly with event sourcing. I also find it to be a handy way of thinking of raw and derived data, rather than trying to perfect a normalized data model to serve all purposes.
- dsies 5y ago
  The "reimplementing" part resonates with me 100%. I've also reimplemented this solution many times and every time it has been a pain in the ass. I built https://batch.sh https://batch.sh specifically to address this - not having to reinvent the wheel for storage, search and replay. For some cases, storing your events in a pg database is probably good enough - but if you're planning on storing billions of complex records AND fetching a particular group of them every now and then - it'll get rough and you need a more sophisticated storage system. What storage mechanism did you use? And how did you choose which events to replay?
  - acjohnson55 5y ago
    In our case, we used Postgres. Our event volume was quite small, and we needed strict consistency (i.e. all participants in the auction see the same state). So we stored the log of events for an auction lot as a JSON blob. A new command was processed by taking a row-level lock (SELECT FOR UPDATE) on the lot's row, validating the command, and then persisting the events. Then we'd broadcast the new derived state to all interested parties. All command processing and state queries required us to read and process the whole event log. But this was fairly cheap, because we're talking maybe a couple dozen events per lot. To optimize, we might have considered serializing core aspects of the derived state, to use as a snapshot. But this wasn't necessary. Batch looks pretty cool! I'll keep that in mind next time I'm considering reinventing the wheel :)
Yuioup 5y ago
Event Sourcing advocates: How do you deal with cleaning up events, especially with respect to the GDPR. I inherited an application with 24M+ events and I have no idea how to go about it.
- ComodoHacker 5y ago
  Not an advocate, but you just run the script to clean the data (as ordinary events). It's not really different than handling regular backups. You don't wipe them, just guard them well until they expire.
  - dgb23 5y ago
    Does it comply when I cant see the data but it is still there?
- Fiahil 5y ago
  Two paths : encrypt customer data in payloads with a customer-specific key that you can throw away at will ; OR ; allow some events to rewrite their history / stream.
- oweiler 5y ago
  In Kafka, you can use log-compacted topics, which only keep the last snapshot of a key, and allows deletions via tombstones.
- const314 5y ago
  We don't delete events, just replace sensitive info in events with the specific string marker (by direct update in DB) and then update projections. In this case, the system remains consistent because all the related aggregates and events are still there. You can safely rebuild all the projections without issues and see these string markers instead of deleted info.
  - dsies 5y ago
    ^ 100% this. Do not delete events. They are your source of truth - you shouldn't really even modify them but stripping out PII is "alright". Re 24M+ records: create a batch runner that goes through "jobs" to perform stripping/cleaning tasks. To store state (and to organize cleaners), use a distributed store such as etcd - that way you can bookmark where you were at in the cleaning process.
Xenoamorphous 5y ago
I’m surprised to see positive experiences with event sourcing in the comments, I had the impression that in older posts about event sourcing there were horror stories, and I think even Martin Fowler said in most cases he’d seen it didn’t work. Or maybe he was referring specifically to CQRS, but isn’t that used most of the time along event sourcing?
- pyrale 5y ago
  Honestly, you should take negative comments about technology on HN (and elsewhere) with a pinch of salt. By reading HN, I have learned that the language I earn a living with is unfit for industry use. I also frequently encounter claims that the stuff I do is impossible, etc. From what I have seen however, industry is much more technologically diverse than people imagine. Event sourcing is an amazing tool. The failing projects are usually the ones that want to use it without acknowledging that it offers different tradeoffs from more traditional data modeling/storing solutions, and not adapting to meet different ups and downs. There are also many projects that want to use event sourcing, but actually end up picking up event sourcing, reactive programming and a host of frameworks or even languages to do the second. That's a lot of technical baggage to learn at once, and people making the switch in a production context with tight deadlines and without deep pockets to hire experienced people usually don't fare so well.
conceptme 5y ago
Interesting to see a lot of positive comments about CQRS + Event sourcing, I only had one experience with it and it wasn't good after about 1.5 year of development it needed to be rewritten without ever being in production. The whole thing was super complex to understand, the event flow that could be triggered made it hard to understand what was going on and migrations where very difficult to do correctly. Notifying frontend with "eventual consistency" was pretty terrible to.
TobiasA 5y ago
How do people handle versioning events, like adding/removing/changing properties? It seems like your application would always need to be aware of all previous event versions to be able to replay all events.
- vgel 5y ago
  Where I work (meetdandy.com, digital dental lab), we do a combination of: For small changes like adding a field, we make the property on the underlying event optional, but require it in the event dispatch function. That allows old events to be missing the field, but type-checking still requires it in the codebase going forward. It does require a special case to handle the missing field, however. For larger changes to events, we do migrations. We store events in Postgres, so we just use TypeORM migrations. I haven’t had to do one of these myself, but I’ve heard secondhand that it’s not too bad. We don’t have very many of these, regardless.
danielskogly 5y ago
This article[0] by Arkwrite continues where Martin Fowler left off, and does a good job of explaining what Event Sourcing is, complete with illustrations and stick-men. [0] https://arkwright.github.io/event-sourcing.html https://arkwright.github.io/event-sourcing.html
ilitirit 5y ago
The biggest problem I've encountered in places that adopt Event Sourcing is using it as a hammer. Not every piece of information needs to be event sourced, and neither is it a good fit for every use case.
rphln 5y ago
I'm experimenting with this for client-server design in game development, to ensure that the clients are in sync with the server: since the only way to change the game state is through events, I (as a developer) am unable to forget to implement some notification for something that happens. This also gets me match replays as an extra. I came across the Update Monad [1] while researching how to use Event Sourcing in functional languages: it basically has the same guarantees of only changing the state through recorded events. Beware though, as it can be quite slow if implemented naively. [1]: https://chrispenner.ca/posts/update-monad https://chrispenner.ca/posts/update-monad
whalesalad 5y ago
The adage “when you have a hammer, everything starts to look like a nail” comes to mind. Event sourcing is a very powerful concept, but it’s very easy to use inappropriately. If you can’t map your business/domain problem very easily into the paradigm, it is worth pausing and reconsidering. Don’t get stuck maintaining a system you never should have built in the first place.
dsies 5y ago
I'm late to this party but I'll chime in anyway - I love event sourcing. So much so that I actually built a company around it (and got into YC S20) - https://batch.sh https://batch.sh Event sourcing is not easy but the benefits are huge - not only from a software architecture perspective but from a systems perspective - you gain a whole lot of additional platform reliability. There are roughly 3 pillars to event sourcing that have to be built (most of the time) from scratch - storing events (forever), reading stored events, replaying stored events. Those are the 3 things I've built at several companies and it is always a huge barrier to someone taking up the pattern. Of course there are more things under the hood, but those are foundational pieces that will make the whole experience much better. Would love to chat with folks who want to nerd out over this stuff :-)

105 points by gjvc 5y ago

swyx 5y ago

We debated the precise meaning of Event Sourcing when I was writing my Temporal.io explainer (https://www.swyx.io/why-temporal/ https://www.swyx.io/why-temporal/) Basically the source of the debate - to do "proper" event sourcing, do you need to rerun the computations each time you roll back/forward, or is it enough to simply restore state?

bluepizza 5y ago
Hello. I love Temporal, it is an amazing solution. I have been pushing it pretty strongly at my workplace, but you guys need to make that web interface better. I lose way too many people at the demo stage. Next year on the Q1 I'll be able to push for a full demo again. Hope we get traction this time. It's great tech, and a pleasure to use.
- swyx 5y ago
  i'm literally working on it as we speak :) hear you loud and clear. my top 2 priorities rn are TypeScript SDK and new Web UI.
dtornow 5y ago
I was part of that debate, I remember a rather interesting point of discussion: Is the main operation "apply" or "dedup" Apply seems to be the common notion of event sourcing: There is a function apply that takes a state an event and yields a new state. Then, starting in an init state and iteratively applying the entire event history, boom, latest state restored. Dedup has a lot of charm though: Run and rerun your code, if that step of your code is executed for the first time (no corresponding event in the event history) execute the step and store its result as an event in the history, however, if that step of your code is executed for the second, third time (there is a corresponding event in the event history) do not execute the step and return its result from the event in the history. The Haskell Workflow Package (https://hackage.haskell.org/package/Workflow https://hackage.haskell.org/package/Workflow) is a good example Temporal follows the second approach, so "proper" Event Sourcing? You be the judge :)
- btown 5y ago
  This may be how Temporal works, but I imagine the way you ask "is there a corresponding event in the event history" is that you tag every "derived event" in the system with a hash of both the state and code used to create it? So if you change the code, you (eventually?) invalidate all events derived from it? This of course would be incredibly reliant on the code not being able to access anything except the state, and have no side effects - so if you want to say "if X, fetch Y" you need to have something looking at derived state from the "if X" part and putting the results of "fetch Y" into the event stream, thus causing the state relevant to the code to change, and invalidating the need for something to "fetch Y" a second time? There's something incredibly clean and enticing about this, but also incredibly scary to implement correctly at scale!
  - mfateev 5y ago
    Temporal takes much simpler approach to updating code while a program is running. For any code update, it keeps both old and the new version of the code. Then it uses the old code to replay already recorded events which allows to reconstruct the program state and takes the new code path when it is executed for the first time. Eventually the old code is removed once all the executions that used it are completed.
acjohnson55 5y ago
Can't remember how I first encountered your online presence, but I'm pretty sure I've followed you on Twitter for a while. Didn't realize you had joined Temporal. I've followed Temporal for a while, although I haven't yet had the chance to use it professionally. Still, I'm fairly convinced it's The Future, especially as it becomes obvious how to combine it with UX and data layer (right now, it's a bit unclear to me how much of UX state and permanent storage should live in Temporal). If I were building a green field business with automation at its core, I'd very strongly consider picking Temporal.
- swyx 5y ago
  thank you! pls let us know if we can help with those questions when its time to evaluate us seriously.
pyrale 5y ago
That's a technical detail imo, as long as your implementation guarantees the outcomes are identical. The reason some people replay events is that it's a simple and easy way to restore previous state.

bob1029 5y ago

This is the direction I want to take our product. Running RCA on one of our transactions is really difficult because we only ever store the current state of affairs. I am still struggling with the grain on which I would want to source events. The engineer in me says its safest to go low-level and source key-value, so it doesnt matter if the biz model changes. But, this would reduce fidelity on the other side in terms of scanning for specific events of importance (e.g. SetKeyValue vs SetCustomerEmail).

dharmaturtle 5y ago
I would argue that there is no such thing as "the business model". There may be a projected rolled up summary state... but that's only one possible view out of infinitely many. For example, one possible "business model" could be "customers who have changed their emails in the past month". You shouldn't favor any particular view as being the favored "business model"... unless you're doing snapshots, in which case everything goes out the window. Basically, I vote for `SetCustomerEmail`.
- Thiez 5y ago
  Events have already happened. The idiomatic name would be `CustomerEmailSet` or perhaps just `EmailSet` or `EmailChanged`.
timClicks 5y ago
Start with very granular events. They can be aggregated later if needed, but the reverse isn't possible.
- rphln 5y ago
  Isn't it the other way around? If you have an “AB” event, it's fairly trivial to go through the log and split it into “A then B”. If you have “A then C then B” for whatever reason and then realize that you need to aggregate “A” and “B”, that is much harder.
loldot_ 5y ago

erikerikson 5y ago

(2005)

dang 5y ago
Added. Thanks!

charles_f 5y ago

Posted 5 times but never really discussed https://hn.algolia.com/?q=https%3A%2F%2Fmartinfowler.com%2FeaaDev%2FEventSourcing.html https://hn.algolia.com/?q=https%3A%2F%2Fmartinfowler.com%2Fe...

gjvc 5y ago
Was this time though...

criticaltinker 5y ago

> Reversal is the most straightforward when the event is cast in the form of a difference. An example of this would be "add $10 to Martin's account" as opposed to "set Martin's account to $110". In the former case I can reverse by just subtracting $10, but in the latter case I don't have enough information to recreate the past value of the account. I've used the event sourcing pattern with great success in the context of an exceptionally complex web application for my day job. The front-end would create an 'event' for every action the user took. This allowed us to seamlessly implement client-side undo/redo functionality with surprisingly little effort, a market leading feature our users really appreciated. One key insight we had was the necessity of capturing the state change as a difference, as Martin explains. This turned out to be tremendously powerful. We were able to build a variety of other features on top including detailed analytics of how users were using our app, and session replay for training and debugging, among other things.

sbazerque 5y ago
from 10000ft away that has the feeling of op-based CRDTs
- fennecfoxen 5y ago
  More or less.
- swagasaurus-rex 5y ago
  Event sourcing can be represented as a chronological list of changes, replayed so that any list index represents a state. How CRDTs work is different in different implementations. Some CRDTs use a timestamp to order all changes, thus making it essentially identical to an event sourced list. Timestamp based ordering may not actually be the most ideal, especially for low trust environments where the timestamp can be spoofed. Some apps might not last-write wins. In this case, timestamp largely becomes meaningless because there has to be some other way to resolve conflicts due to multiple clients making changes to the same piece of data. Some apps want one clear winner, other apps might want the two inputs to be merged, others (like git) explicitly ask the developer to resolve the conflict. In this way CRDTs can diverge significantly from an ordered list of changes.

dharmaturtle 5y ago

Event sourcing is really good for building occasionally connected applications (e.g. phone app). Makes conflict resolution so much easier than, say two relational databases. Basically it follows the git model - distributed systems are each their own sources of truth, and allows for (relatively) simple syncing of state. When two distributed systems conflict, well... you need to write some kind of merge algorithm that may or may not ask for user input. But at least the accidental complexity is significantly reduced.

dminor 5y ago

I often find myself wishing for some sort of small scale durable log, so that this pattern was easy to implement in a small app backend without standing up something like Kafka.

emerongi 5y ago
I recently refactored a small set of services to use events instead of depending directly on eachother. I initially had Kafka in mind, but it would've been an absolute overkill. A "simple Kafka" would definitely be nice for these cases. I ended up using Redis Streams [0], which was good enough for my small-scale need. We already had Redis in our stack too, so it was a very simple integration. [0] https://redis.io/topics/streams-intro https://redis.io/topics/streams-intro
uDontKnowMe 5y ago
Have you considered rabbitmq or nats?
dharmaturtle 5y ago
What's wrong with EventStore? https://www.eventstore.com/ https://www.eventstore.com/
- dminor 5y ago
  Nice! Hadn't heard of it, will check it out.
  - dharmaturtle 5y ago
    They have a channel in this slack that's pretty good for getting answers to questions: https://github.com/ddd-cqrs-es/slack-community https://github.com/ddd-cqrs-es/slack-community
dimitrios1 5y ago
try jetstream
5y ago

gjvc 5y ago

TDD stole much, if not all, of the airtime Event Sourcing should have received.

vmurthy 5y ago

As a techie-turned-PM , I found this pattern really useful when discussing specs with different stakeholders. Business stakeholders understand their business as a series of events (Customer walked in, customer bought X , customer checked out etc) so the user stories become a lot more relatable to both developers and business folks. I vaguely remember events being discussed in "Domain modelling made functional" , too

gjvc 5y ago
Very much agree. I had a memorable conversation with a PM a short time before the pandemic about this very subject. She was enthused about how it would make her work more closely aligned that of developers.

swagasaurus-rex 5y ago

I recently built https://github.com/siriusastrebe/jsynchronous https://github.com/siriusastrebe/jsynchronous which uses event sourcing to synchronize javascript variables on the server with connected browsers. You can also replay states using a special “rewind” mode, a core advantage of event sourcing.

acjohnson55 5y ago

Event sourcing is a game changer. Everywhere I have worked has ended up reimplementing it in some form, because it is irresistible to incorporate history into your data model. My most thorough use of event sourcing was in a real-time art auction system, which combined in-person and online bidding. An operator had to input the actions in the room into the system and bid on behalf of online bidders. Event sourcing allowed us to model what actually happened for each auction lot. It also made it possible to model undos and cancellations in really slick ways. Command-query response segregation (CQRS) is also a powerful concept, and it combines elegantly with event sourcing. I also find it to be a handy way of thinking of raw and derived data, rather than trying to perfect a normalized data model to serve all purposes.

dsies 5y ago
The "reimplementing" part resonates with me 100%. I've also reimplemented this solution many times and every time it has been a pain in the ass. I built https://batch.sh https://batch.sh specifically to address this - not having to reinvent the wheel for storage, search and replay. For some cases, storing your events in a pg database is probably good enough - but if you're planning on storing billions of complex records AND fetching a particular group of them every now and then - it'll get rough and you need a more sophisticated storage system. What storage mechanism did you use? And how did you choose which events to replay?
- acjohnson55 5y ago
  In our case, we used Postgres. Our event volume was quite small, and we needed strict consistency (i.e. all participants in the auction see the same state). So we stored the log of events for an auction lot as a JSON blob. A new command was processed by taking a row-level lock (SELECT FOR UPDATE) on the lot's row, validating the command, and then persisting the events. Then we'd broadcast the new derived state to all interested parties. All command processing and state queries required us to read and process the whole event log. But this was fairly cheap, because we're talking maybe a couple dozen events per lot. To optimize, we might have considered serializing core aspects of the derived state, to use as a snapshot. But this wasn't necessary. Batch looks pretty cool! I'll keep that in mind next time I'm considering reinventing the wheel :)

Yuioup 5y ago

Event Sourcing advocates: How do you deal with cleaning up events, especially with respect to the GDPR. I inherited an application with 24M+ events and I have no idea how to go about it.

ComodoHacker 5y ago
Not an advocate, but you just run the script to clean the data (as ordinary events). It's not really different than handling regular backups. You don't wipe them, just guard them well until they expire.
- dgb23 5y ago
  Does it comply when I cant see the data but it is still there?
Fiahil 5y ago
Two paths : encrypt customer data in payloads with a customer-specific key that you can throw away at will ; OR ; allow some events to rewrite their history / stream.
oweiler 5y ago
In Kafka, you can use log-compacted topics, which only keep the last snapshot of a key, and allows deletions via tombstones.
const314 5y ago
We don't delete events, just replace sensitive info in events with the specific string marker (by direct update in DB) and then update projections. In this case, the system remains consistent because all the related aggregates and events are still there. You can safely rebuild all the projections without issues and see these string markers instead of deleted info.
- dsies 5y ago
  ^ 100% this. Do not delete events. They are your source of truth - you shouldn't really even modify them but stripping out PII is "alright". Re 24M+ records: create a batch runner that goes through "jobs" to perform stripping/cleaning tasks. To store state (and to organize cleaners), use a distributed store such as etcd - that way you can bookmark where you were at in the cleaning process.

Xenoamorphous 5y ago

I’m surprised to see positive experiences with event sourcing in the comments, I had the impression that in older posts about event sourcing there were horror stories, and I think even Martin Fowler said in most cases he’d seen it didn’t work. Or maybe he was referring specifically to CQRS, but isn’t that used most of the time along event sourcing?

pyrale 5y ago
Honestly, you should take negative comments about technology on HN (and elsewhere) with a pinch of salt. By reading HN, I have learned that the language I earn a living with is unfit for industry use. I also frequently encounter claims that the stuff I do is impossible, etc. From what I have seen however, industry is much more technologically diverse than people imagine. Event sourcing is an amazing tool. The failing projects are usually the ones that want to use it without acknowledging that it offers different tradeoffs from more traditional data modeling/storing solutions, and not adapting to meet different ups and downs. There are also many projects that want to use event sourcing, but actually end up picking up event sourcing, reactive programming and a host of frameworks or even languages to do the second. That's a lot of technical baggage to learn at once, and people making the switch in a production context with tight deadlines and without deep pockets to hire experienced people usually don't fare so well.

conceptme 5y ago

Interesting to see a lot of positive comments about CQRS + Event sourcing, I only had one experience with it and it wasn't good after about 1.5 year of development it needed to be rewritten without ever being in production. The whole thing was super complex to understand, the event flow that could be triggered made it hard to understand what was going on and migrations where very difficult to do correctly. Notifying frontend with "eventual consistency" was pretty terrible to.

TobiasA 5y ago

How do people handle versioning events, like adding/removing/changing properties? It seems like your application would always need to be aware of all previous event versions to be able to replay all events.

vgel 5y ago
Where I work (meetdandy.com, digital dental lab), we do a combination of: For small changes like adding a field, we make the property on the underlying event optional, but require it in the event dispatch function. That allows old events to be missing the field, but type-checking still requires it in the codebase going forward. It does require a special case to handle the missing field, however. For larger changes to events, we do migrations. We store events in Postgres, so we just use TypeORM migrations. I haven’t had to do one of these myself, but I’ve heard secondhand that it’s not too bad. We don’t have very many of these, regardless.

danielskogly 5y ago

This article[0] by Arkwrite continues where Martin Fowler left off, and does a good job of explaining what Event Sourcing is, complete with illustrations and stick-men. [0] https://arkwright.github.io/event-sourcing.html https://arkwright.github.io/event-sourcing.html

ilitirit 5y ago

The biggest problem I've encountered in places that adopt Event Sourcing is using it as a hammer. Not every piece of information needs to be event sourced, and neither is it a good fit for every use case.

rphln 5y ago

I'm experimenting with this for client-server design in game development, to ensure that the clients are in sync with the server: since the only way to change the game state is through events, I (as a developer) am unable to forget to implement some notification for something that happens. This also gets me match replays as an extra. I came across the Update Monad [1] while researching how to use Event Sourcing in functional languages: it basically has the same guarantees of only changing the state through recorded events. Beware though, as it can be quite slow if implemented naively. [1]: https://chrispenner.ca/posts/update-monad https://chrispenner.ca/posts/update-monad

whalesalad 5y ago

The adage “when you have a hammer, everything starts to look like a nail” comes to mind. Event sourcing is a very powerful concept, but it’s very easy to use inappropriately. If you can’t map your business/domain problem very easily into the paradigm, it is worth pausing and reconsidering. Don’t get stuck maintaining a system you never should have built in the first place.

dsies 5y ago

I'm late to this party but I'll chime in anyway - I love event sourcing. So much so that I actually built a company around it (and got into YC S20) - https://batch.sh https://batch.sh Event sourcing is not easy but the benefits are huge - not only from a software architecture perspective but from a systems perspective - you gain a whole lot of additional platform reliability. There are roughly 3 pillars to event sourcing that have to be built (most of the time) from scratch - storing events (forever), reading stored events, replaying stored events. Those are the 3 things I've built at several companies and it is always a huge barrier to someone taking up the pattern. Of course there are more things under the hood, but those are foundational pieces that will make the whole experience much better. Would love to chat with folks who want to nerd out over this stuff :-)