4 ms·

Event sourcing is a game changer. Everywhere I have worked has ended up reimplementing it in some form, because it is irresistible to incorporate history into y

Event sourcing is a game changer. Everywhere I have worked has ended up reimplementing it in some form, because it is irresistible to incorporate history into your data model. My most thorough use of event sourcing was in a real-time art auction system, which combined in-person and online bidding. An operator had to input the actions in the room into the system and bid on behalf of online bidders. Event sourcing allowed us to model what actually happened for each auction lot. It also made it possible to model undos and cancellations in really slick ways. Command-query response segregation (CQRS) is also a powerful concept, and it combines elegantly with event sourcing. I also find it to be a handy way of thinking of raw and derived data, rather than trying to perfect a normalized data model to serve all purposes.

dsies 5y ago
The "reimplementing" part resonates with me 100%. I've also reimplemented this solution many times and every time it has been a pain in the ass. I built https://batch.sh https://batch.sh specifically to address this - not having to reinvent the wheel for storage, search and replay. For some cases, storing your events in a pg database is probably good enough - but if you're planning on storing billions of complex records AND fetching a particular group of them every now and then - it'll get rough and you need a more sophisticated storage system. What storage mechanism did you use? And how did you choose which events to replay?
- acjohnson55 5y ago
  In our case, we used Postgres. Our event volume was quite small, and we needed strict consistency (i.e. all participants in the auction see the same state). So we stored the log of events for an auction lot as a JSON blob. A new command was processed by taking a row-level lock (SELECT FOR UPDATE) on the lot's row, validating the command, and then persisting the events. Then we'd broadcast the new derived state to all interested parties. All command processing and state queries required us to read and process the whole event log. But this was fairly cheap, because we're talking maybe a couple dozen events per lot. To optimize, we might have considered serializing core aspects of the derived state, to use as a snapshot. But this wasn't necessary. Batch looks pretty cool! I'll keep that in mind next time I'm considering reinventing the wheel :)

dsies 5y ago

The "reimplementing" part resonates with me 100%. I've also reimplemented this solution many times and every time it has been a pain in the ass. I built https://batch.sh https://batch.sh specifically to address this - not having to reinvent the wheel for storage, search and replay. For some cases, storing your events in a pg database is probably good enough - but if you're planning on storing billions of complex records AND fetching a particular group of them every now and then - it'll get rough and you need a more sophisticated storage system. What storage mechanism did you use? And how did you choose which events to replay?

acjohnson55 5y ago
In our case, we used Postgres. Our event volume was quite small, and we needed strict consistency (i.e. all participants in the auction see the same state). So we stored the log of events for an auction lot as a JSON blob. A new command was processed by taking a row-level lock (SELECT FOR UPDATE) on the lot's row, validating the command, and then persisting the events. Then we'd broadcast the new derived state to all interested parties. All command processing and state queries required us to read and process the whole event log. But this was fairly cheap, because we're talking maybe a couple dozen events per lot. To optimize, we might have considered serializing core aspects of the derived state, to use as a snapshot. But this wasn't necessary. Batch looks pretty cool! I'll keep that in mind next time I'm considering reinventing the wheel :)