6 ms·

I feel bad for anyone who got sucked into event sourcing for general purpose applications. Like... I'm burning a candle for you right now. The event stream stu

I feel bad for anyone who got sucked into event sourcing for general purpose applications. Like... I'm burning a candle for you right now. The event stream stuff makes sense in some specialized use cases (LMAX?) but for general purpose stuff? Oof, no. EDIT: Also, don't hate me folks, but any article that references Fowler should be looked at with just a little bit of suspicion. Not hating the guy but he's blown quite a bit of hot air in his time.

nuclearnice1 5y ago
What makes event sourcing poor for general purpose stuff?
- Spartan-S63 5y ago
  Not OP, but the number of variants in the events makes the cardinality of the state of the system extremely large. Probably too large to be able to reason about fully. It makes weird bugs in the event schema likely impossible to debug should they occur.
  - 88913527 5y ago
    This is exactly why 2-way data binding fell out of favor in modern UI development. Describing the problem as "the cardinality of the state of the system extremely large" is a succinct way of putting it. The non-linear increase in possible state quickly makes testing an uphill battle. And it's over when you can't have a comprehensive testing strategy, forget about it.
    - Spivak 5y ago
      Eh, two-way binding is still alive and well it’s just implemented explicitly on top of one-way bindings. It seems most developers find this easier to reason about but the relationship is still there. Like if all you’re doing is using the event to take the value from the UI and shove it into the model with setState and re-render then you made a two-way binding. Wouldn’t it be helpful if you could tell React to just do this for you and save you the boilerplate? Boom you invented ng-model.
    - square_usual 5y ago
      2-way bindings are still alive and well in the big "next gen" web frameworks, Vue and Svelte.
  - btown 5y ago
    Can this be mitigated by ensuring that historical/legacy events are all transformed to more modern equivalents? Similar in theory to https://stripe.com/blog/api-versioning https://stripe.com/blog/api-versioning ? Then you at least have a chain of testable transformations and a reduced cardinality for the actual interactions that reflects the cardinality of all the ways your business users actually use your product today. Of course, this can be lossy, but is it any more lossy than the non-event-sourced CRUD system which is essentially transforming historical events into "overwrite these fields" events that can only be used for their original purpose? To me, the reason not to use event sourcing is more that the tooling and developer experience is just really poor - and so decisions like the above, while valuable to think through directly, rapidly become overwhelming. Visualizations of system state are only as good as the code you bring yourself. I'm hopeful that people building tools on top of things like https://materialize.com https://materialize.com will start to change this though!
- _3u10 5y ago
  Event sourced systems are actually great, as long as they aren’t branded as such. Most of the time it’s totally stupid and the state of the system isn’t reproducible anyway unless the event code is never modified.
- mr_tristan 5y ago
  This seemed like a great rundown on why event sourcing is complicated: https://medium.com/@hugo.oliveira.rocha/what-they-dont-tell-you-about-event-sourcing-6afc23c69e9a https://medium.com/@hugo.oliveira.rocha/what-they-dont-tell-...
- bsder 5y ago
  Event sourcing is decent when you ingest a lot of things that are mostly one way and immutable once they come into your system. If you have bidirectional communication, you wind up contorting yourself into a pretzel to bury "state" into your event streams.
- wrwatson 5y ago
  The first rule of data modelling is that you should model only what you care about. A good model allows you to store and retrieve what you need to meet your requirements. Maybe you care about all the events that have ever happened on your incredible journey to reach a list of the customers you have today. Or maybe it would be sufficient to have been mutating a customers table all along. If that is sufficient, then it would be faster and more straightforward to have done so.
  - mistahenry 5y ago
    This is why I always like to have a `status_history` or `audit` table. You don't have to have event sourcing to still want a trail of who changed what in the system when.
- holoduke 5y ago
  Extra layer of complexity. More development time. More tech dependencies. Don't use it when not needed.
the_duke 5y ago
Event sourcing could be a great architecture for general purpose applications - if we had good frameworks and specialized event stores built for this purpose. Some features that would be needed: * Automatic , schema based event and aggregate versioning * A combined event and aggregate data store with ACID guarantees, schema validation and indexes * event migrations (so you don't have to carry around old event versions forever) * Event stream migrations that allow re-writing streams for purging data, gdpr cleanup, etc * Copy on write based stream clones that can be used for testing with production data As it stands, such a solution is nowhere to be seen. So you have to cobble together a working system with lots of custom work , which often ends in a complete mess.
- jordanbeiber 5y ago
  Look at Zeebe[1] for example. If you declare the business process you want to support - either in code or in BPMN, you're automatically mapping out the events that are relevant to the business. Having a system that takes care of executing this, and tying the pieces of this workflow together is one example where ES can shine IMHO. [1]https://docs.camunda.io/docs/product-manuals/zeebe/technical-concepts/internal-processing https://docs.camunda.io/docs/product-manuals/zeebe/technical...
dmitryminkovsky 5y ago
I second this wholeheartedly because this happened to me. I'm the kind of person who learns by my making my own mistakes (as opposed to learning from others', which is what smart people do)—and boy, did I learn a lot from the mistake of trying to build a general purpose application with event sourcing. Theoretically, event sourcing is the way all applications should be built. I was first taken by this methodology by Martin Kleppmann's talk "Turning the Database Inside Out" (2014)[0]. Looking back, I think I had always subconsciously felt like there was something wrong with mutating state in place. You lose information. You lose accountability. After all, "accountants don't use erasers," and Martin's talk validated many of my obsessive-compulsive insecurities about data. As Charity Majors points out on Twitter [1]: > ...while storing raw events may be "expensive" compared to any single optimization, all optimizations can be derived from raw events. The reverse is impossible. And that's true. The table-stream duality [2][3] means that if you store events, you can later project those events into the exact perfect table for any use-case. But if you transform those events into a projection and store that projection, well, whatever table you store is data you have. You wind up having just an aggregate. Unfortunately, the technical costs you incur when you attempt to store events instead of tabular data are tremendous. The problems and questions that arise are basically endless. Tons of things you wouldn't even think about with a relational database become complicated design decisions [4]. If there were tooling and frameworks that made this easy, it would be one thing. But, there are not! Kafka Streams is very cool. If you already have a running business with well-developed models and a large team, event sourcing with Kafka Streams could possibly go quite well. But if you need to get an MVP off the ground, it's just not what you need. What you need is PostgreSQL, perhaps with something like Hasura or postgraphile in front of it. What you need is a lightning-fast way to iterate and flesh out your idea. If that's you, and you're reading this, and you're smart: learn from me—event sourcing: don't do it! [0] https://www.youtube.com/watch?v=fU9hR3kiOK0 https://www.youtube.com/watch?v=fU9hR3kiOK0 [1] https://twitter.com/mipsytipsy/status/1115537408705957888 https://twitter.com/mipsytipsy/status/1115537408705957888 [2] https://docs.confluent.io/platform/current/streams/concepts.html#duality-of-streams-and-tables https://docs.confluent.io/platform/current/streams/concepts.... [3] https://news.ycombinator.com/item?id=23207123 https://news.ycombinator.com/item?id=23207123 [4] https://softwareengineering.stackexchange.com/questions/380196/should-i-store-an-event-stream-or-just-state-for-a-frequently-changing-entity https://softwareengineering.stackexchange.com/questions/3801...
deleted 5y ago
[deleted]
copperx 5y ago
> any article that references Fowler should be looked at with just a little bit of suspicion. Not hating the guy but he's blown quite a bit of hot air in his time. I've just started reading some Fowler stuff. What hot air are you referring to? I would like to keep that in mind.
- blowski 5y ago
  People over-apply his patterns. Some patterns that are useful in niche edge cases suddenly become a ‘best practice standard’, sprayed anywhere and everywhere. My rule of thumb is that you should only use his patterns if you could afford to hire him to code it for you.
  - tunesmith 5y ago
    One of those books has a rule buried in the middle somewhere that I was was in 60-point text on the first page of chapter one. I've taken to calling it Rule Zero: Don't implement a pattern unless it actually solves a problem.
    - blowski 5y ago
      Exactly. The problem is not Martin Fowler himself, but those who pick up his books and ignore those warnings.
goostavos 5y ago
Count me in the long line of suckers :( Probably one of the largest technical blunders of my career (or at least the largest so far!) was pulling a team down into ES hell on a greenfield project. Turns out, growing an event sourced application is fantastically hard[0]. Ours eventually became a twisted opaque mess which was terrifying to modify due to (a) the sheer volume of code it took to do so, and (b) eventually losing control over who read which events and where as the team grew. [0] https://chriskiehl.com/article/event-sourcing-is-hard https://chriskiehl.com/article/event-sourcing-is-hard
beaker52 5y ago
I'm working with a team that's in the process of decomissioning a previous teams attempt at ES. Even the new team wanted to do ES, whilst staring at the remains of the previous attempt. Luckily they haven't. Usually arguments go "we need an audit log so ES is a natural fit and we'll be able to iterate quicker because we can just re-project when we get things wrong, oh and testing is really straightforward too because we can just...". You can have an audit log without ES. You're also going to hate versioning events. And navigating your tangled nest of events that you accidentally accumulated in the name of agile isn't the nirvana you were hoping for. I believe it's better to have a system that responds to the requirements of here and now, leaving history behind, learning from mistakes going forward and accepting that yesterday is done. In my experience, in growing software systems, you're more likely to have not to have even had the data/event at all, than to not have misappreciated what you already had. So event logs feel more like a bunch of baggage ready to trip teams up than a wealth of value.
ryandvm 5y ago
> any article that references Fowler should be looked at with just a little bit of suspicion Indeed. I don't doubt he's a very clever guy, but all of my most miserable gigs involved phalanxes of architecture astronauts endlessly bike-shedding every design session with the platitudes of Fowler et al.

nuclearnice1 5y ago

What makes event sourcing poor for general purpose stuff?

Spartan-S63 5y ago
Not OP, but the number of variants in the events makes the cardinality of the state of the system extremely large. Probably too large to be able to reason about fully. It makes weird bugs in the event schema likely impossible to debug should they occur.
- 88913527 5y ago
  This is exactly why 2-way data binding fell out of favor in modern UI development. Describing the problem as "the cardinality of the state of the system extremely large" is a succinct way of putting it. The non-linear increase in possible state quickly makes testing an uphill battle. And it's over when you can't have a comprehensive testing strategy, forget about it.
  - Spivak 5y ago
    Eh, two-way binding is still alive and well it’s just implemented explicitly on top of one-way bindings. It seems most developers find this easier to reason about but the relationship is still there. Like if all you’re doing is using the event to take the value from the UI and shove it into the model with setState and re-render then you made a two-way binding. Wouldn’t it be helpful if you could tell React to just do this for you and save you the boilerplate? Boom you invented ng-model.
  - square_usual 5y ago
    2-way bindings are still alive and well in the big "next gen" web frameworks, Vue and Svelte.
- btown 5y ago
  Can this be mitigated by ensuring that historical/legacy events are all transformed to more modern equivalents? Similar in theory to https://stripe.com/blog/api-versioning https://stripe.com/blog/api-versioning ? Then you at least have a chain of testable transformations and a reduced cardinality for the actual interactions that reflects the cardinality of all the ways your business users actually use your product today. Of course, this can be lossy, but is it any more lossy than the non-event-sourced CRUD system which is essentially transforming historical events into "overwrite these fields" events that can only be used for their original purpose? To me, the reason not to use event sourcing is more that the tooling and developer experience is just really poor - and so decisions like the above, while valuable to think through directly, rapidly become overwhelming. Visualizations of system state are only as good as the code you bring yourself. I'm hopeful that people building tools on top of things like https://materialize.com https://materialize.com will start to change this though!
_3u10 5y ago
Event sourced systems are actually great, as long as they aren’t branded as such. Most of the time it’s totally stupid and the state of the system isn’t reproducible anyway unless the event code is never modified.
mr_tristan 5y ago
This seemed like a great rundown on why event sourcing is complicated: https://medium.com/@hugo.oliveira.rocha/what-they-dont-tell-you-about-event-sourcing-6afc23c69e9a https://medium.com/@hugo.oliveira.rocha/what-they-dont-tell-...
bsder 5y ago
Event sourcing is decent when you ingest a lot of things that are mostly one way and immutable once they come into your system. If you have bidirectional communication, you wind up contorting yourself into a pretzel to bury "state" into your event streams.
wrwatson 5y ago
The first rule of data modelling is that you should model only what you care about. A good model allows you to store and retrieve what you need to meet your requirements. Maybe you care about all the events that have ever happened on your incredible journey to reach a list of the customers you have today. Or maybe it would be sufficient to have been mutating a customers table all along. If that is sufficient, then it would be faster and more straightforward to have done so.
- mistahenry 5y ago
  This is why I always like to have a `status_history` or `audit` table. You don't have to have event sourcing to still want a trail of who changed what in the system when.
holoduke 5y ago
Extra layer of complexity. More development time. More tech dependencies. Don't use it when not needed.

the_duke 5y ago

Event sourcing could be a great architecture for general purpose applications - if we had good frameworks and specialized event stores built for this purpose. Some features that would be needed: * Automatic , schema based event and aggregate versioning * A combined event and aggregate data store with ACID guarantees, schema validation and indexes * event migrations (so you don't have to carry around old event versions forever) * Event stream migrations that allow re-writing streams for purging data, gdpr cleanup, etc * Copy on write based stream clones that can be used for testing with production data As it stands, such a solution is nowhere to be seen. So you have to cobble together a working system with lots of custom work , which often ends in a complete mess.

jordanbeiber 5y ago
Look at Zeebe[1] for example. If you declare the business process you want to support - either in code or in BPMN, you're automatically mapping out the events that are relevant to the business. Having a system that takes care of executing this, and tying the pieces of this workflow together is one example where ES can shine IMHO. [1]https://docs.camunda.io/docs/product-manuals/zeebe/technical-concepts/internal-processing https://docs.camunda.io/docs/product-manuals/zeebe/technical...

dmitryminkovsky 5y ago

I second this wholeheartedly because this happened to me. I'm the kind of person who learns by my making my own mistakes (as opposed to learning from others', which is what smart people do)—and boy, did I learn a lot from the mistake of trying to build a general purpose application with event sourcing. Theoretically, event sourcing is the way all applications should be built. I was first taken by this methodology by Martin Kleppmann's talk "Turning the Database Inside Out" (2014)[0]. Looking back, I think I had always subconsciously felt like there was something wrong with mutating state in place. You lose information. You lose accountability. After all, "accountants don't use erasers," and Martin's talk validated many of my obsessive-compulsive insecurities about data. As Charity Majors points out on Twitter [1]: > ...while storing raw events may be "expensive" compared to any single optimization, all optimizations can be derived from raw events. The reverse is impossible. And that's true. The table-stream duality [2][3] means that if you store events, you can later project those events into the exact perfect table for any use-case. But if you transform those events into a projection and store that projection, well, whatever table you store is data you have. You wind up having just an aggregate. Unfortunately, the technical costs you incur when you attempt to store events instead of tabular data are tremendous. The problems and questions that arise are basically endless. Tons of things you wouldn't even think about with a relational database become complicated design decisions [4]. If there were tooling and frameworks that made this easy, it would be one thing. But, there are not! Kafka Streams is very cool. If you already have a running business with well-developed models and a large team, event sourcing with Kafka Streams could possibly go quite well. But if you need to get an MVP off the ground, it's just not what you need. What you need is PostgreSQL, perhaps with something like Hasura or postgraphile in front of it. What you need is a lightning-fast way to iterate and flesh out your idea. If that's you, and you're reading this, and you're smart: learn from me—event sourcing: don't do it! [0] https://www.youtube.com/watch?v=fU9hR3kiOK0 https://www.youtube.com/watch?v=fU9hR3kiOK0 [1] https://twitter.com/mipsytipsy/status/1115537408705957888 https://twitter.com/mipsytipsy/status/1115537408705957888 [2] https://docs.confluent.io/platform/current/streams/concepts.html#duality-of-streams-and-tables https://docs.confluent.io/platform/current/streams/concepts.... [3] https://news.ycombinator.com/item?id=23207123 https://news.ycombinator.com/item?id=23207123 [4] https://softwareengineering.stackexchange.com/questions/380196/should-i-store-an-event-stream-or-just-state-for-a-frequently-changing-entity https://softwareengineering.stackexchange.com/questions/3801...

deleted 5y ago

[deleted]

copperx 5y ago

> any article that references Fowler should be looked at with just a little bit of suspicion. Not hating the guy but he's blown quite a bit of hot air in his time. I've just started reading some Fowler stuff. What hot air are you referring to? I would like to keep that in mind.

blowski 5y ago
People over-apply his patterns. Some patterns that are useful in niche edge cases suddenly become a ‘best practice standard’, sprayed anywhere and everywhere. My rule of thumb is that you should only use his patterns if you could afford to hire him to code it for you.
- tunesmith 5y ago
  One of those books has a rule buried in the middle somewhere that I was was in 60-point text on the first page of chapter one. I've taken to calling it Rule Zero: Don't implement a pattern unless it actually solves a problem.
  - blowski 5y ago
    Exactly. The problem is not Martin Fowler himself, but those who pick up his books and ignore those warnings.

goostavos 5y ago

Count me in the long line of suckers :( Probably one of the largest technical blunders of my career (or at least the largest so far!) was pulling a team down into ES hell on a greenfield project. Turns out, growing an event sourced application is fantastically hard[0]. Ours eventually became a twisted opaque mess which was terrifying to modify due to (a) the sheer volume of code it took to do so, and (b) eventually losing control over who read which events and where as the team grew. [0] https://chriskiehl.com/article/event-sourcing-is-hard https://chriskiehl.com/article/event-sourcing-is-hard

beaker52 5y ago

I'm working with a team that's in the process of decomissioning a previous teams attempt at ES. Even the new team wanted to do ES, whilst staring at the remains of the previous attempt. Luckily they haven't. Usually arguments go "we need an audit log so ES is a natural fit and we'll be able to iterate quicker because we can just re-project when we get things wrong, oh and testing is really straightforward too because we can just...". You can have an audit log without ES. You're also going to hate versioning events. And navigating your tangled nest of events that you accidentally accumulated in the name of agile isn't the nirvana you were hoping for. I believe it's better to have a system that responds to the requirements of here and now, leaving history behind, learning from mistakes going forward and accepting that yesterday is done. In my experience, in growing software systems, you're more likely to have not to have even had the data/event at all, than to not have misappreciated what you already had. So event logs feel more like a bunch of baggage ready to trip teams up than a wealth of value.

ryandvm 5y ago

> any article that references Fowler should be looked at with just a little bit of suspicion Indeed. I don't doubt he's a very clever guy, but all of my most miserable gigs involved phalanxes of architecture astronauts endlessly bike-shedding every design session with the platitudes of Fowler et al.