4 ms·
I second this wholeheartedly because this happened to me. I'm the kind of person who learns by my making my own mistakes (as opposed to learning from others',
by dmitryminkovsky 5y ago
I second this wholeheartedly because this happened to me.
I'm the kind of person who learns by my making my own mistakes (as opposed to learning from others', which is what smart people do)—and boy, did I learn a lot from the mistake of trying to build a general purpose application with event sourcing.
Theoretically, event sourcing is the way all applications should be built. I was first taken by this methodology by Martin Kleppmann's talk "Turning the Database Inside Out" (2014)[0]. Looking back, I think I had always subconsciously felt like there was something wrong with mutating state in place. You lose information. You lose accountability. After all, "accountants don't use erasers," and Martin's talk validated many of my obsessive-compulsive insecurities about data. As Charity Majors points out on Twitter [1]:
> ...while storing raw events may be "expensive" compared to any single optimization, all optimizations can be derived from raw events. The reverse is impossible.
And that's true. The table-stream duality [2][3] means that if you store events, you can later project those events into the exact perfect table for any use-case. But if you transform those events into a projection and store that projection, well, whatever table you store is data you have. You wind up having just an aggregate.
Unfortunately, the technical costs you incur when you attempt to store events instead of tabular data are tremendous. The problems and questions that arise are basically endless. Tons of things you wouldn't even think about with a relational database become complicated design decisions [4]. If there were tooling and frameworks that made this easy, it would be one thing. But, there are not!
Kafka Streams is very cool. If you already have a running business with well-developed models and a large team, event sourcing with Kafka Streams could possibly go quite well.
But if you need to get an MVP off the ground, it's just not what you need. What you need is PostgreSQL, perhaps with something like Hasura or postgraphile in front of it. What you need is a lightning-fast way to iterate and flesh out your idea. If that's you, and you're reading this, and you're smart: learn from me—event sourcing: don't do it!
[0] https://www.youtube.com/watch?v=fU9hR3kiOK0 https://www.youtube.com/watch?v=fU9hR3kiOK0
[1] https://twitter.com/mipsytipsy/status/1115537408705957888 https://twitter.com/mipsytipsy/status/1115537408705957888
[2] https://docs.confluent.io/platform/current/streams/concepts.html#duality-of-streams-and-tables https://docs.confluent.io/platform/current/streams/concepts....
[3] https://news.ycombinator.com/item?id=23207123 https://news.ycombinator.com/item?id=23207123
[4] https://softwareengineering.stackexchange.com/questions/380196/should-i-store-an-event-stream-or-just-state-for-a-frequently-changing-entity https://softwareengineering.stackexchange.com/questions/3801...
- jordanbeiber 5y agoI believe ES has it’s uses in discreet systems doing their thing for you. For me ES makes perfect sense in systems handing a workflow for you. Products like Zeebe[1] and Ubers Temporal[2] comes to mind. These systems basically becomes your business transaction log and as such ES is just a great fit. (I have no idea whether ES is used in temporal though… never looked at the code) [1] https://github.com/camunda-cloud/zeebe https://github.com/camunda-cloud/zeebe [2] https://temporal.io/ https://temporal.io/
- swyx 5y agohi! I work at Temporal. ftr our founders left Uber 2 years ago and Temporal is an independent startup (that is very much hiring) now. Short answer is yes, we do, but it's abstracted away so you get the benefits (complete observability, retry/resume from failure) without the downsides (handwriting event sourcing logic and storage). Long answer... it depends what your definition of event sourcing is. We shared some of our internal debate here: https://news.ycombinator.com/item?id=28149159 https://news.ycombinator.com/item?id=28149159 Event sourcing is an impt implementation detail for us, but it is not something we push onto our users. Still they benefit from it anyway. very much agree with your statement. happy to take followup qtns :)
- jareklupinski 5y ago> learning from others', which is what smart people do my grandfather had the same words :) > What you need is PostgreSQL, perhaps with something like Hasura or postgraphile in front of it. What you need is a lightning-fast way to iterate and flesh out your idea. optimizing my team's incoming stream stack atm, event sourcing seems so enticing on paper...