15 ms·

Event Sourcing Is Hard (2019)

286 points by swyx 5y ago

deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
lampe3 5y ago
This is from 2018.
- gefhfff 5y ago
  Is it easy now?
  - oweiler 5y ago
    The technologies (Kafka, Kafka Connect, Kafka Streams, whatever you use) have stabilized somewhat but Event Sourced systems are as complex as ever.
    - tkubacki 5y ago
      Kafka and friends can be used in reactive pattern paradigm ( understood as here https://www.youtube.com/watch?v=eRxLfUIMJwk&t https://www.youtube.com/watch?v=eRxLfUIMJwk&t ) - only partially overlaping ES idea, which in general has much more sense to me.
  - lampe3 5y ago
    HN usually adds the year in the title if its is not from the current year.
wongarsu 5y ago
Previous discussion (2019): https://news.ycombinator.com/item?id=19072850 https://news.ycombinator.com/item?id=19072850 Has anything changed since then? Is there new tooling that resolves some of the pain points, or new techniques that change the picture?
- brianmcc 5y ago
  A whole new bunch of people for whom the article hits home maybe. In any case - it's a new one to me, and thanks for the link to earlier discussion, v interesting!
alecco 5y ago
Event Sourcing sucks. I blame OOP abstractions taken to an extreme. https://vvvvalvalval.github.io/posts/2018-11-12-datomic-event-sourcing-without-the-hassle.html https://vvvvalvalval.github.io/posts/2018-11-12-datomic-even...
- misja111 5y ago
  Strange that you blame this on OOP. I see event sourcing being used particularly often in FP environments. Because of course the model of streaming immutable data fits very well into the FP paradigm.
- valw 5y ago
  I wrote the blog post you cited (thanks!) but I disagree with both statements: that is not what is meant in the article. 1. I don't think Event Sourcing sucks - I think we are lacking accessible technology for supporting it. 2. For most difficulties encountered in Event Sourcing, I would rather blame distribution taken to the extreme.
bob1029 5y ago
> The idea of a keeping a central log against which multiple services can subscribe and publish is insane. Is event sourcing about storing 1 authoritative, immutable, serialized narrative of events, or is it about tying together a pile of microservices? I think the part where it goes wrong is where we try to subscribe to these events (i.e. push). This adds delivery semantics to the mix and forces a lot more constraints at vendor selection time. Being notified that something occurred is not part of event sourcing in my mind (though it is clearly one logical extenstion).
- politician 5y ago
  I've built these systems. One major problem is that both scenarios are called event sourcing by different folks. The Confluent (Kafka) people will tell you that ES is a streaming approach while the EventStore folks will tell you that it's about replays, aggregates and immutability. Different blogs refer to wildly different use cases with the same name and almost no one states their assumptions up front. Don't get me started on explaining snapshots, why you don't need snapshots until replays take 1000ms, why replays won't take 1000ms until the topics are thousands of events deep, why the team won't believe that this isn't a problem, and why you will definitely need snapshots if you keep scaling.
  - EdwardDiego 5y ago
    From my experience with stateful streaming apps, snapshots are life.
deleted 5y ago
[deleted]
nocture 5y ago
Lots of hype about event sourcing and CQRS, usually in combination. 99% won't need it. Has the potential to ruin your team and even company unless you have a well-above-average engineering team.
- datavirtue 5y ago
  This. Why do people try unproven tech in a business? A well-above-average engineering team isn't going to go for this. I would leave before this shit show even got started. The problem should be begging for this pattern if it fits. Like those mentioning the use of ES for high speed stock trading, for example. One of my unwritten requirements for any system is being able to drop in inexperienced devs for maintenance...after I leave. Most people are likely to shoot themselves in the foot by not understanding (or even being aware of) database isolation levels much less grokking an event pipeline.
Yuioup 5y ago
I'm busy rewriting a project to go back to regular SQL. Whoever thought Event Sourcing was a good idea should retire from software development.
- quickthrower2 5y ago
  They probably have the money to retire from the consulting fees
kmac_ 5y ago
ES is really great, but just for a narrow set of service types and when it's applied locally in a closed bussines domain. But as a global system architecture, nope, I tried that once and it only brought more costs.
- stepbeek 5y ago
  100% agree. I used it for position management software in equities and it fit like a glove. Effectively, we had a stream of transactions coming from upstream that we aggregated into a cache. Being able to replay events, or step through to figure out why something had gone wrong was terrific. I then tried to apply this to a conventional request-response web service. It did not fit well. Everything was immediately significantly harder with only a marginal gain. My rule fo thumb now is to use event sourcing when doing otherwise would be difficult. A justification based off of an improvement, rather than one solving an obvious problem, isn't worth the extra effort.
FeepingCreature 5y ago
Silly question: isn't the way you're supposed to handle the observer pattern thing to be that you can follow processing chains using correlation IDs? Ie. the initial event of a dataflow gets tagged with a unique ID and then there's some database magic that lets you track that ID through the system? Like, I can see where that would go wrong once you merge multiple events together, but that seems a different failure point than described here. And while we're at it, the whole point of eventsourcing is that they're not supposed to model "internal" process state? They're supposed to be the ground source of truth; whatever the process has to do to mangle that into a model it can work with should be no skin off the events' back. The rest seems very, very correct to my experience. Especially not knowing the pain points until you're past the toy level.
- ggm 5y ago
  "Database magic" is worrisome
  - FeepingCreature 5y ago
    Eh, you could do it manually by hanging a projection on $all and republishing to streams per correlation id. But it's cheaper to let the db do it internally.
    - ggm 5y ago
      So it's not a strict dependency, more in the spirit probably of an optimisation: "here's how you own it, but most people let the substrate do it for you..." type thing.
- adamauckland 5y ago
  Yes, Correlation ID/Activity ID/Request ID, you need something that ties the event together. I think Event Sourcing is an amazing model for certain scenarios. Often, a hybrid approach can be better.
akiselev 5y ago
I've worked with several event sourcing systems and was even seduced into implementing one out of sheer hubris once. These problems are ever present in every ES project I've had the misfortune of coming into contact with. It doesn't even mention the worst part that comes afterwards, when you realize after all of that pain that it is only used by a single person in the company to generate a noncritical report comparing a meaningless KPI that could have been manually done in four hours by a different intern every quarter. By the time the tooling is up and running enough to make a stable system, no one will trust it enough to use ES's landmark features except for a few developers still coming off the koolaid. 99% of the time when ES sounds like a good idea, the answer is to just use Postgres. Use wal2json and subscribe to the WAL stream - if you really need time travel or audit logs or whatever, they'll be much cheaper to implement using WAL. If you need something more enterprisey to sell to the VP-suite, use Debezium. Event sourcing sounds so awesome in theory and it is used to great effect in many demanding applications (like Postgres! WAL = event sourcing with fewer steps) but it's just too complex for non-infrastructure software without resources measured in the 10s or 100s of man years.
- bsder 5y ago
  > 99% of the time when ES sounds like a good idea, the answer is to just use Postgres. I like some of the things in Event Sourcing very much. However, whenever I ask "Why not just use PostgreSQL as the event store until I'm 5 orders of magnitude bigger?" I never seem to get a really good answer.
- EdwardDiego 5y ago
  Debezium isn't that enterprisey, unless you buy support from Red Hat ;)
- brainwipe 5y ago
  My experience also. I've worked with 3 different ones and I think the domain fit was good for only one of them. In that one domain model, the fit was spot on because that domain was just a bunch of events and every question you asked of the domain model was really saying "create me an object with the events between time X and Y". That was it tho, all other domains I've worked on would not suit ES - even though they would suit CQRS without the ES piece.
geocar 5y ago
I think one major problem is that "Event Sourcing" can mean subtly different things to different people. > The idea of a keeping a central log against which multiple services can subscribe and publish is insane. This really doesn't mean "Event Sourcing" to me, it sounds like enterprises that have decided Event Sourcing == Kafka (or some cloud-hosted IOT-branded variant) and treat the central broker/coordinators/confluent-cloud crap as "the central log" To me, the fundamental idea is that changes are recorded in a meaningful format, not the result of changes; What I mean by "meaningful" is important: I don't think SQL statement replication (as implemented by popular SQL databases) constitutes "event sourcing", because the only part of the application that typically consumes this is the SQL database, but if you're storing JSON in a log (or better: some useful binary format) then you're probably doing event sourcing. With this (perhaps broad) view of Event Sourcing, I would say I have been building all of my database applications for the last 30 years as "event sourcing", and I've never had any trouble building applications (even large ones!) using this model: It lends it self to an excellent user-experience simply by letting the UI "in on it", and the article seems to recognise this: > Event sourcing needs the UI side to play along If your user makes a change, and for whatever reason that change takes time to do, it is better to tell the user you're going to do it and then do it rather than wait for the HTTP request (or TCP message or whatever) to "complete" the transaction online because network always fails and bad error messages (the default!) cause users to do stupid things. But when you use Google Cloud's UI, you can see how nice this can be: You make a change, you can look in the event log to see your changes, you can see if they've been processed or not, and you can examine and share the results simply by sharing the results pages (instead of having to make screenshots or other ghastly things). I think for many applications this is worth it, but there aren't good tools for event sourcing (in my opinion) so it may be for a lot of applications (especially the ones I don't work on) the juice just isn't worth the squeeze -- but to me, this suggests value in developing those tools, not in ignoring this amazing thing.
- dm3 5y ago
  > I think one major problem is that "Event Sourcing" can mean subtly different things to different people. You're right. Here's a list of posts[1] about Event Streaming which claim to be about Event Sourcing. The discussed article is in the list. [1]: https://github.com/oskardudycz/EventSourcing.NetCore#1319-this-is-not-event-sourcing-but-event-streaming https://github.com/oskardudycz/EventSourcing.NetCore#1319-th...
the-alt-one 5y ago
The big issue, to me, with concepts such as event sourcing is that they are fundamentally presented as software engineering concepts. You get a nice picture showcasing the overall structure and a story or two of how succesful this is, but we never discuss computer science matters. I want to know things such as "what safety properties does this system afford me?", "how is it abstractly modelled?", "what global invariants must I uphold for this system to work?". These are missing in regular conversation with other engineers, and I miss this kind of more formal reasoning. Ultimately, this kind of discussion should form the basis for the answers to questions of value to the business. Bugs arise and are solved in these informal systems, but I rarely hear people ask "how do we never get this particular class of bug again?". These questions are kept for researchers (either academic or corporation), never to reach the regular software developer.
- Kototama 5y ago
  How does that differ from other architecture patterns?
  - the-alt-one 5y ago
    This isn't unique to event sourcing, I definitely took the opportunity to soapbox.
- hcarvalhoalves 5y ago
  Event sourcing is reifying a state machine transitions explicitly (in data). So the properties will depend on how the state machine is modeled. If you don’t have strong ordering guarantees, it also means your state machine has to accept transitions from/to any state (eg comutative). Otherwise if ordering is important, your event log must have some ordering built-in (exactly once semantics) or you implement re-ordering and replay as past events arrive (a branching model).
- Floegipoky 5y ago
  What is driving the adoption of an event sourced model other than a need for properties/guarantees that a reasonably normalized relational model can't provide? If the engineering team isn't talking about tradeoffs and the formal properties of the system, that seems like an indictment of the local engineering culture rather than anything to do with event sourcing.
justusw 5y ago
Is Event Sourcing compatible with GDPR?
- ddek 5y ago
  The issue with GDPR is right to erasure, which is tricky in conventional systems. But in ES systems, naively implemented, it is not possible. If you are writing PII in plain text, an immutable log won't let you remove it. OTOH I have two options: store PII in a separate database, and just delete or nullify when requested; or use per-user encryption, and delete the key.
  - EdwardDiego 5y ago
    Yeah, for GDPR compliance when you're using Kafka, the key is to use, well keys, and log compaction. Then you can retrieve PII for a GDPR if needed. Or, remove it entirely by submitting a "tombstone" record with key -> null, then log compaction removes it. Which basically means, don't mix PII into transnational data.
  - lucian1900 5y ago
    The GDPR concept is "personal data", partly since it doesn't just refer to ways to identify a person. PII is a similar but different US concept.
- dairyleia 5y ago
  An Event-sourced system can be compliant with GDPR, with crypto shredding: https://www.eventstore.com/blog/protecting-sensitive-data-in-event-sourced-systems-with-crypto-shredding-1 https://www.eventstore.com/blog/protecting-sensitive-data-in...
bruth 5y ago
The title here is being overlooked, its not _terrible_, its _hard_. Also this quote from the article: > The bulk of these probably fall under "he obviously didn't understand X," or "you should never do Y!" in which case you would be absolutely right. The point of this is that I didn't understand the drawbacks or pain points until I'd gotten past the "toy" stage. Most of what they discussed was indeed not great, but that was due to lack of understanding. I don't think anyone who does or promotes ES says it's _easy_. However, I would argue it _conceptually_ simple to understand (herein lies the fallacy people make simple != easy). Like any pattern or tech that you are new too, it's going to be hard to build and operate in production (beyond the toy). Everyone takes for granted the decades we've had great tooling and operational patterns for state-oriented databases (mostly RDBMS'). And to be clear, event sourcing is "persistence strategy". What is mostly shown/discussed in the article is "event streaming" along with CQRS. They concepts are largely conflated. Once you understand, you won't be bamboozled. Not to throw Confluent under the bus, but they have unfortunately written or promoted many articles for "event sourcing" use cases and this has confused everyone. You _can't_ do event sourcing with Kafka. It doesn't support optimistic concurrency control at the key level (where as something like EventStoreDB does natively or even NATS now with JetStream). Kafka if fantastic for what it is designed to do.. event streaming.
- geoduck14 5y ago
  I've never heard of the term Event Sourcing. I AM familiar with Event Streaming. Some of what this article talks about surprised me - and I think it is because I don't fully understand the difference between Event Sourcing and Event Streaming. Is this article a fine summary?: https://developer.confluent.io/learn-kafka/event-sourcing/event-sourcing-vs-event-streaming/ https://developer.confluent.io/learn-kafka/event-sourcing/ev...
  - bruth 5y ago
    I have not watched that clip, but as I said above, Confluent isn't a good resource for defining this term since Kafka cannot be used to do ES. I would suggest an article like: https://domaincentric.net/blog/eventstoredb-vs-kafka https://domaincentric.net/blog/eventstoredb-vs-kafka Also refer to the list of great resources @oskar_dudycz posted in another comment: https://github.com/oskardudycz/EventSourcing.NetCore#1319-this-is-not-event-sourcing-but-event-streaming https://github.com/oskardudycz/EventSourcing.NetCore#1319-th...
EdwardDiego 5y ago
Most common issue I've ever encountered with event sourcing is that devs submit events that when replayed, lead to invalid states. And if you're using an immutable log (Kafka/Pulsar/Liftbridge etc.) fixing those mistakes is hard. And given you're basically recording interim states of a state machine, how do you verify that event Z being submitted after event Y is a valid state transition? Validating a record against a schema is trivial. But validating three consecutive events against the state machine? I'm not aware of a good way to do that cheaply. I've done it before using apps that sat in the data pipeline, and maintained state to reject invalid transitions, but that can end up being GiB of state, then you have to persist that across restarts or horizontal scaling. It's all doable, but it costs. Is the event streaming paradigm actually worth the cost? I haven't been convinced yet. But, I'm a big fan of streaming DB change events for auditing purposes - but those are coming from software that already enforces valid state changes. So maybe the answer is - every event has to be submitted via Postgres.
- Lhiw 5y ago
  > Most common issue I've ever encountered with event sourcing is that devs submit events that when replayed, lead to invalid states. This doesn't sound like event sourcing. It's likely your events aren't events or someone changed what your event handlers do at a later date instead of releasing a new version of the event.
- randolorian 5y ago
  Seems like a fundamental misunderstanding to me. Commands get validated via your aggregate, not events. Events are not allowed to be rejected. Ever. Because they already happened.
  - EdwardDiego 5y ago
    Thinking that event sourcing and CQRS are synonyms is a fundamental misunderstanding also. Anyway, pedantry aside, my point remains thus - it's hard to prevent invalid state transitions being submitted to an event sourcing datastore, and when devs can submit events directly, then these invalid transitions will occur.
Liron 5y ago
Ok but what if the event log was a decentralized database with money built in, would that help?
Liron 5y ago
I think there's a middle option between (1) building your application up from a low-level stream-of-events data model and (2) having your application full of logic to snap from one high-level state to another without a good data-layer audit trail: I'm talking about indexes. Yes, the index part of your database is amazing, because it constantly snaps itself into various states, and you know exactly why: because it needs to maintain itself in the same deterministic relationship with your models. I think that if databases would support more complex and composable indexes, like providing the ability to index join queries, then application programmers would be more nimble and bug-free working with their data. I've written more about this here: https://lironshapira.medium.com/data-denormalization-is-broken-7b697352f405 https://lironshapira.medium.com/data-denormalization-is-brok...
ilitirit 5y ago
One thing that I've learned from Event Sourcing is that it makes bad engineering teams and processes worse.
oskar_dudycz 5y ago
As this article pops out again, I'd like to the point that although it may have some valid points, those points are not about Event Sourcing. What's expressed in the article is the Event Streaming or Event-Driven approach. So when events are not the source of truth etc. All of the event stores that I know supports strong consistency on appends, optimistic concurrency. Many guarantee global ordering. Some help in idempotency. All of that helps to reduce those issues. In Event Sourcing, events are the state. So the flow looks like that: 1. You get the events from the stream (that represents all the facts that happened for the entity), 2. You apply it one by one in the order of appearance to build the current state, 3. You run the business logic and, as a result, create the event, 4. You append a new event. You don't need to cache the write model state anywhere, as the state is in events. Event Sourcing has its issues and troubles, but the stuff described in the article apply to tools like Kafka, etc. They're not tools for Event Sourcing but Event Streaming. They're designed to move things from one place to another, not to be used as durable databases. I've made a list of articles like that: https://github.com/oskardudycz/EventSourcing.NetCore#this-is-not-event-sourcing https://github.com/oskardudycz/EventSourcing.NetCore#this-is... I'll repeat it again: this is not Event Sourcing.
- kreetx 5y ago
  BTW, does there happen to be a discord channel for this topic? ;)
  - oskar_dudycz 5y ago
    There is a Slack channel: https://join.slack.com/t/ddd-cqrs-es/shared_invite/zt-m3vf3alt-S3L~YUoIV88wekj6wSNrUQ https://join.slack.com/t/ddd-cqrs-es/shared_invite/zt-m3vf3a...
- nhumrich 5y ago
  I re-read the article after this comment, and I would have to disagree. The article does never attempt to explain "what" recent sourcing means to them, so it's hard to know for sure. However, they do mention populating state from an event log which contains meaningless events, so I have to assume they _are_ talking about event sourcing. What leads you to believe they are taking about just streaming, and not sourcing?
DrBazza 5y ago
I've worked on a few event sourcing systems, and the only ones that have been truly successful are the ones where the system was written from scratch. Trying to retrofit systems never seemed to quite work as expected. Also, the "from scratch" systems, seemed conceptually easier to not only understand, but extend, because you were forced to write code in an idiomatic and consistent way. Everything was an event (albeit, a "command" and an "event" confirming 'execution' of the "command").
- bob1029 5y ago
  I am currently looking at a new iteration of our product that uses an event log as the principal data store. I agree that if we start from "zero" and don't even provide the option to build things the wrong way it would go smoothly. "everything is an event" is very powerful if its not just some add-on policy. For us, the entire working set would be able to live in memory. Larger objects would be left as pointers to the actual data in the log. Snapshots of working set would be taken on a daily basis in hopes of keeping recovery under ~5 minutes during service restarts or updates.
  - DrBazza 5y ago
    > Snapshots of working set would be taken on a daily basis in hopes of keeping recovery under ~5 minutes during service restarts or updates. Exactly this. We took hourly snapshots (for a trading system), and could restart the system in a second or so. Another nice side effect of an event driven system is that you start to think about what other data you can put in there. And then you realise you don't necessarily need a database any more, which makes the architecture of the system simpler. There are so many benefits working with a data-driven system. Once data is in the log/ledger/whatever you call it, it's immutable, which leads to parallelism if you want it. Ensuring the system is idempotent also means you get (easier) failover as well.
    - bob1029 5y ago
      > And then you realise you don't necessarily need a database any more This is precisely where we want to be. We are already exceptionally minimal when it comes to database usage - SQLite is all we use today. If we are in a place where 100% of the business facts go to an immutable/append-only log, we can handle the entire persistence mechanism with a simple filestream write command. Not having to worry about mutating prior data solves a large portion of database engine problems automagically.
xg15 5y ago
> The idea of a keeping a central log against which multiple services can subscribe and publish is insane > In effect, the raw event stream subscription setup kills the ability to locally reason about the boundaries of a service. Under "normal" development flows, you operate within the safe, cozy little walls which make up your service. I wonder if this isn't more a case of the "cozy little walls" turning out to be a lie. If your business requirements are that this user action has to trigger those half-dozen barely related background tasks, then your design will always be messy and complex, no matter if you use service calls, event queues, polling or whatever else. In such a situation, I think a better question would be which design lets you most easily reason about the entire system - e.g., what is the dependency structure, how is the causal chain organized, etc. Of course there are also "secondary" concerns, such as: Which design lets you make changes to the system easily, can you still easily develop and test things in isolation that can be isolated, etc. I think services and events actually have some upsides here (but I still may be in the "seduced" phase...) I think this calls more for a better API/dependency management (and treating events as full-fledged APIs!) and a good tracing solution which lets you follow an action through different services (and event queues). > You're free to make choices about implementation and storage and then, when you're ready, deal with how those things get exposed to the outside world. It's one of the core benefits of "services". However, when people are reaching into your data store and reading your events directly, that 'black box' property goes out the window Again, I would absolutely treat events just as much as a public API as REST. Only because the causal realtionship is inversed doesn't mean you can throw away schemas, versioning, dependency tracking etc.
adamauckland 5y ago
Event Sourcing is absolutely brilliant if you want to build an offline first/distributed system which will become eventually consistent. A good example would be a tree inspection application where jobs can be pushed out to mobile inspectors who might not have phone signal but can still collect the data. Once synchronised, views are simply additive. More data can be added to cases simply by adding events. I would absolutely not use ES for a hotel booking system because that data needs to be as accurate as possible in the moment and we've had record locking techniques for over 40 years. Seems a lot of people want to try ES without really considering whether it maps to the real-world domain of the problem they're trying to solve.
- picardo 5y ago
  > A good example would be a tree inspection application where jobs can be pushed out to mobile inspectors who might not have phone signal but can still collect the data. Once synchronised, views are simply additive. More data can be added to cases simply by adding events. This isn't a great use case for Event Souring either. An inspector must be in front of the tree to inspect it, right? And they can see each other inspecting the tree.... Why would several inspectors inspect the same tree? Assuming they inspect the same tree at different times, you can sync offline data at the row level, and let the last update win, provided that they're entering the same data.
  - adamauckland 5y ago
    Where did I say multiple inspectors are inspecting a tree? There might indeed be multiple people dealing with a tree. The tree owner, an inspector, a tree surgeon, the local authorities (if the trees are protected). Synchronising data at row level when offline becomes online is extremely hard to get right and reliable. With the Event Sourcing model, every new piece of data gets thrown into the bucket , with each new piece of data the view becomes bigger.
    - picardo 5y ago
      > Synchronising data at row level when offline becomes online is extremely hard to get right and reliable. Why? Are the users in different time zones? If the time on their devices are synchronized, why is it hard to get right? I'm not trying to pick an argument with you, but trying to figure out if you've considered all the possible alternatives.
invisible 5y ago
Adding something like change-data-capture to a system and using that for some of the purposes that event sourcing touts seems like it goes a lot further and doesn't require making a complex, hard-to-understand system. Having a CDC system to publish events and record those can give the biggest benefits and sit alongside a much simpler system. A downside is that the real-time feature goes away.
oftenwrong 5y ago
This matches closely with my experience working at a company that was mostly built on event-sourcing. I would add a few things, some of which are touched upon in the article: - The general problem with event-sourcing (at small scale) is that it forces you to have to think about and handle many things that can mostly be ignored with a more typical persistence approach. This overhead of complexity was one of the factors that (IMO) killed a company I worked for. Event-sourcing feels relatively "raw" in this way. However, this also provided a valuable learning experience as a developer. - I'm curious about better ways to do event-sourcing. For example, using Datomic is apparently like a well-thought-out, pre-made event-sourcing system: https://vvvvalvalval.github.io/posts/2018-11-12-datomic-event-sourcing-without-the-hassle.html https://vvvvalvalval.github.io/posts/2018-11-12-datomic-even... - If you intend to embark on an event-sourcing journey, read Versioning in an Event- Sourced System https://leanpub.com/esversioning/read https://leanpub.com/esversioning/read , even if only to better understand the complexity of this one aspect. I agree with the author of this submission that it is hard to fully grasp the complexities of event-sourcing without trying it, and hitting many road bumps along the way, since it's not an especially well-trodden path. - Being able to redefine the interpretation of an event as the system evolves feels like a superpower at times. Similarly, being able to bring a new part of the system online, and have it based on a rich history of everything ever done in the system, is very cool. One of the major benefits of event-sourcing is how adaptable it is in systems that change over time. - Beware CQRS and eventual consistency. This adds a lot of complexity and is not a prerequisite for event-sourcing. It's also telling that ordinarily-simple problems to solve, like preventing duplicate username registrations, are hand-waved away by CQRS proponents: https://web.archive.org/web/20191101010824/http://codebetter.com/gregyoung/2010/08/12/eventual-consistency-and-set-validation/ https://web.archive.org/web/20191101010824/http://codebetter... . You will face this class of problems, and you may not be able to hand-wave it away so easily.
valw 5y ago
I submit that Event Sourcing (or at least something very close to it) can be easy, once you've removed some technology locks. I've seen it happen with Datomic (https://vvvvalvalval.github.io/posts/2018-11-12-datomic-event-sourcing-without-the-hassle.html https://vvvvalvalval.github.io/posts/2018-11-12-datomic-even...), but it can probably happen similarly with other tech, such as XTDB (https://xtdb.com/ https://xtdb.com/).
ryanmarsh 5y ago
This is a balanced article and I appreciate the author’s treatment of the challenges. Part of my work is helping enterprise shops with n-tier architectures decouple services to allow for more resilience end to end. Everything the author states has crept up, some of the issues I consider the “nominal cost of resilience and independence”. Other issues are easily mitigated, such as the yak shaving required to get off the ground. Serverless solves the infra/config challenges. If an enterprise shop isn’t open to serverless I have no interest in discussing event sourcing or CQRS with them. It’s too much work otherwise. For shops that are being crushed by fluctuations in concurrency, challenges with the uptime and p50 of partner teams, don’t have much of a choice but to go event sourced, optimistic, and eventually consistent. I’m certain a great enterprise IT shop with great leadership could mitigate the challenges of service growth and instability with some clever optimizations of their current stacks but unfortunately I have to meet a shop that is uniformly competent. Event sourcing helps with the competency gap problem. If every team’s data is on the outside my team can work around your lead time for changes or general ops incompetence. If inventory is down, fuck you I’m still taking orders. The extra work of maintaining a projection of your inventory events makes good business sense because time kills all deals.
acjohnson55 5y ago
This article is great. I architected an ES system once. It was absolutely the right call, but we had to work through nearly every one of those problems. Regardless of whether you do it well or not, engineers will find it cumbersome to work with and think that it's overengineered.
swyx 5y ago
I work at Temporal[1], where we solve a bunch of these problems for you by cleanly abstracting the critical event sourcing bits under our SDKs. I think everyone who tries event sourcing eventually rolls their own framework or takes one of the shelf, and Temporal is the most battle tested one I've ever found. The reason I posted this was that I found it helpful in prepping for a workshop I was giving today - recording is not up yet but slides are here https://docs.google.com/presentation/d/1mAPYXNmRdDKepXylB91F2uqFyHm8rG12K2PMRHxoUfM/edit?usp=sharing https://docs.google.com/presentation/d/1mAPYXNmRdDKepXylB91F... 1: https://temporal.io https://temporal.io
pm24601 5y ago
Also showed up on hacker news in 2019 https://news.ycombinator.com/item?id=19072850 https://news.ycombinator.com/item?id=19072850
mixmastamyk 5y ago
What the hell is event sourcing? I read the first half of the article and gave up. Sounded like one of those distributed log things like kafka or maybe a message broker like rabbit? Yes, could google, but now am discouraged. If you're going to write one of these, at least link to the wikipedia page: https://en.wikipedia.org/wiki/Domain-driven_design#Event_sourcing https://en.wikipedia.org/wiki/Domain-driven_design#Event_sou...
bartread 5y ago
I worked on an event sourced system once, many years ago. I'm not saying I hate the separation of write and read models, and all the challenges that brings (heck, the platform I'm working on now does that, and it's not a big deal), but it was massively bloated and overly complex for a system that would collect some data from users, submit it to a panel of third party services, and display results back to those users. The reasons for going this route were complex, and not entirely clear to me, although I can pick on two: firstly, they'd run into performance and scalability issues with their existing CRUD-ish SQL Server-backed implementation and, secondly, Thoughtworks told them it would be a good idea. I don't want to handwave this away but I've encountered enough insoluble SQL Server performance issues over the years to be confident those issues could have been solved much more simply (and the same is likely true for Oracle, Postgres, et al). I'm not suggesting that intractable performance issues don't exist on relational platforms, just that most times people think they've encountered one, they're mistaken. Would I use event sourcing again? Maybe. But I'd need a really strong use case for it, and I'd certainly kick the tyres on any technical organisation that was using it before I joined.

286 points by swyx 5y ago

deleted 5y ago

[deleted]

deleted 5y ago

[deleted]

lampe3 5y ago

This is from 2018.

gefhfff 5y ago
Is it easy now?
- oweiler 5y ago
  The technologies (Kafka, Kafka Connect, Kafka Streams, whatever you use) have stabilized somewhat but Event Sourced systems are as complex as ever.
  - tkubacki 5y ago
    Kafka and friends can be used in reactive pattern paradigm ( understood as here https://www.youtube.com/watch?v=eRxLfUIMJwk&t https://www.youtube.com/watch?v=eRxLfUIMJwk&t ) - only partially overlaping ES idea, which in general has much more sense to me.
- lampe3 5y ago
  HN usually adds the year in the title if its is not from the current year.

wongarsu 5y ago

Previous discussion (2019): https://news.ycombinator.com/item?id=19072850 https://news.ycombinator.com/item?id=19072850 Has anything changed since then? Is there new tooling that resolves some of the pain points, or new techniques that change the picture?

brianmcc 5y ago
A whole new bunch of people for whom the article hits home maybe. In any case - it's a new one to me, and thanks for the link to earlier discussion, v interesting!

alecco 5y ago

Event Sourcing sucks. I blame OOP abstractions taken to an extreme. https://vvvvalvalval.github.io/posts/2018-11-12-datomic-event-sourcing-without-the-hassle.html https://vvvvalvalval.github.io/posts/2018-11-12-datomic-even...

misja111 5y ago
Strange that you blame this on OOP. I see event sourcing being used particularly often in FP environments. Because of course the model of streaming immutable data fits very well into the FP paradigm.
valw 5y ago
I wrote the blog post you cited (thanks!) but I disagree with both statements: that is not what is meant in the article. 1. I don't think Event Sourcing sucks - I think we are lacking accessible technology for supporting it. 2. For most difficulties encountered in Event Sourcing, I would rather blame distribution taken to the extreme.

bob1029 5y ago

> The idea of a keeping a central log against which multiple services can subscribe and publish is insane. Is event sourcing about storing 1 authoritative, immutable, serialized narrative of events, or is it about tying together a pile of microservices? I think the part where it goes wrong is where we try to subscribe to these events (i.e. push). This adds delivery semantics to the mix and forces a lot more constraints at vendor selection time. Being notified that something occurred is not part of event sourcing in my mind (though it is clearly one logical extenstion).

politician 5y ago
I've built these systems. One major problem is that both scenarios are called event sourcing by different folks. The Confluent (Kafka) people will tell you that ES is a streaming approach while the EventStore folks will tell you that it's about replays, aggregates and immutability. Different blogs refer to wildly different use cases with the same name and almost no one states their assumptions up front. Don't get me started on explaining snapshots, why you don't need snapshots until replays take 1000ms, why replays won't take 1000ms until the topics are thousands of events deep, why the team won't believe that this isn't a problem, and why you will definitely need snapshots if you keep scaling.
- EdwardDiego 5y ago
  From my experience with stateful streaming apps, snapshots are life.

deleted 5y ago

[deleted]

nocture 5y ago

Lots of hype about event sourcing and CQRS, usually in combination. 99% won't need it. Has the potential to ruin your team and even company unless you have a well-above-average engineering team.

datavirtue 5y ago
This. Why do people try unproven tech in a business? A well-above-average engineering team isn't going to go for this. I would leave before this shit show even got started. The problem should be begging for this pattern if it fits. Like those mentioning the use of ES for high speed stock trading, for example. One of my unwritten requirements for any system is being able to drop in inexperienced devs for maintenance...after I leave. Most people are likely to shoot themselves in the foot by not understanding (or even being aware of) database isolation levels much less grokking an event pipeline.

Yuioup 5y ago

I'm busy rewriting a project to go back to regular SQL. Whoever thought Event Sourcing was a good idea should retire from software development.

quickthrower2 5y ago
They probably have the money to retire from the consulting fees

kmac_ 5y ago

ES is really great, but just for a narrow set of service types and when it's applied locally in a closed bussines domain. But as a global system architecture, nope, I tried that once and it only brought more costs.

stepbeek 5y ago
100% agree. I used it for position management software in equities and it fit like a glove. Effectively, we had a stream of transactions coming from upstream that we aggregated into a cache. Being able to replay events, or step through to figure out why something had gone wrong was terrific. I then tried to apply this to a conventional request-response web service. It did not fit well. Everything was immediately significantly harder with only a marginal gain. My rule fo thumb now is to use event sourcing when doing otherwise would be difficult. A justification based off of an improvement, rather than one solving an obvious problem, isn't worth the extra effort.

FeepingCreature 5y ago

Silly question: isn't the way you're supposed to handle the observer pattern thing to be that you can follow processing chains using correlation IDs? Ie. the initial event of a dataflow gets tagged with a unique ID and then there's some database magic that lets you track that ID through the system? Like, I can see where that would go wrong once you merge multiple events together, but that seems a different failure point than described here. And while we're at it, the whole point of eventsourcing is that they're not supposed to model "internal" process state? They're supposed to be the ground source of truth; whatever the process has to do to mangle that into a model it can work with should be no skin off the events' back. The rest seems very, very correct to my experience. Especially not knowing the pain points until you're past the toy level.

ggm 5y ago
"Database magic" is worrisome
- FeepingCreature 5y ago
  Eh, you could do it manually by hanging a projection on $all and republishing to streams per correlation id. But it's cheaper to let the db do it internally.
  - ggm 5y ago
    So it's not a strict dependency, more in the spirit probably of an optimisation: "here's how you own it, but most people let the substrate do it for you..." type thing.
adamauckland 5y ago
Yes, Correlation ID/Activity ID/Request ID, you need something that ties the event together. I think Event Sourcing is an amazing model for certain scenarios. Often, a hybrid approach can be better.

akiselev 5y ago

I've worked with several event sourcing systems and was even seduced into implementing one out of sheer hubris once. These problems are ever present in every ES project I've had the misfortune of coming into contact with. It doesn't even mention the worst part that comes afterwards, when you realize after all of that pain that it is only used by a single person in the company to generate a noncritical report comparing a meaningless KPI that could have been manually done in four hours by a different intern every quarter. By the time the tooling is up and running enough to make a stable system, no one will trust it enough to use ES's landmark features except for a few developers still coming off the koolaid. 99% of the time when ES sounds like a good idea, the answer is to just use Postgres. Use wal2json and subscribe to the WAL stream - if you really need time travel or audit logs or whatever, they'll be much cheaper to implement using WAL. If you need something more enterprisey to sell to the VP-suite, use Debezium. Event sourcing sounds so awesome in theory and it is used to great effect in many demanding applications (like Postgres! WAL = event sourcing with fewer steps) but it's just too complex for non-infrastructure software without resources measured in the 10s or 100s of man years.

bsder 5y ago
> 99% of the time when ES sounds like a good idea, the answer is to just use Postgres. I like some of the things in Event Sourcing very much. However, whenever I ask "Why not just use PostgreSQL as the event store until I'm 5 orders of magnitude bigger?" I never seem to get a really good answer.
EdwardDiego 5y ago
Debezium isn't that enterprisey, unless you buy support from Red Hat ;)
brainwipe 5y ago
My experience also. I've worked with 3 different ones and I think the domain fit was good for only one of them. In that one domain model, the fit was spot on because that domain was just a bunch of events and every question you asked of the domain model was really saying "create me an object with the events between time X and Y". That was it tho, all other domains I've worked on would not suit ES - even though they would suit CQRS without the ES piece.

geocar 5y ago

I think one major problem is that "Event Sourcing" can mean subtly different things to different people. > The idea of a keeping a central log against which multiple services can subscribe and publish is insane. This really doesn't mean "Event Sourcing" to me, it sounds like enterprises that have decided Event Sourcing == Kafka (or some cloud-hosted IOT-branded variant) and treat the central broker/coordinators/confluent-cloud crap as "the central log" To me, the fundamental idea is that changes are recorded in a meaningful format, not the result of changes; What I mean by "meaningful" is important: I don't think SQL statement replication (as implemented by popular SQL databases) constitutes "event sourcing", because the only part of the application that typically consumes this is the SQL database, but if you're storing JSON in a log (or better: some useful binary format) then you're probably doing event sourcing. With this (perhaps broad) view of Event Sourcing, I would say I have been building all of my database applications for the last 30 years as "event sourcing", and I've never had any trouble building applications (even large ones!) using this model: It lends it self to an excellent user-experience simply by letting the UI "in on it", and the article seems to recognise this: > Event sourcing needs the UI side to play along If your user makes a change, and for whatever reason that change takes time to do, it is better to tell the user you're going to do it and then do it rather than wait for the HTTP request (or TCP message or whatever) to "complete" the transaction online because network always fails and bad error messages (the default!) cause users to do stupid things. But when you use Google Cloud's UI, you can see how nice this can be: You make a change, you can look in the event log to see your changes, you can see if they've been processed or not, and you can examine and share the results simply by sharing the results pages (instead of having to make screenshots or other ghastly things). I think for many applications this is worth it, but there aren't good tools for event sourcing (in my opinion) so it may be for a lot of applications (especially the ones I don't work on) the juice just isn't worth the squeeze -- but to me, this suggests value in developing those tools, not in ignoring this amazing thing.

dm3 5y ago
> I think one major problem is that "Event Sourcing" can mean subtly different things to different people. You're right. Here's a list of posts[1] about Event Streaming which claim to be about Event Sourcing. The discussed article is in the list. [1]: https://github.com/oskardudycz/EventSourcing.NetCore#1319-this-is-not-event-sourcing-but-event-streaming https://github.com/oskardudycz/EventSourcing.NetCore#1319-th...

the-alt-one 5y ago

The big issue, to me, with concepts such as event sourcing is that they are fundamentally presented as software engineering concepts. You get a nice picture showcasing the overall structure and a story or two of how succesful this is, but we never discuss computer science matters. I want to know things such as "what safety properties does this system afford me?", "how is it abstractly modelled?", "what global invariants must I uphold for this system to work?". These are missing in regular conversation with other engineers, and I miss this kind of more formal reasoning. Ultimately, this kind of discussion should form the basis for the answers to questions of value to the business. Bugs arise and are solved in these informal systems, but I rarely hear people ask "how do we never get this particular class of bug again?". These questions are kept for researchers (either academic or corporation), never to reach the regular software developer.

Kototama 5y ago
How does that differ from other architecture patterns?
- the-alt-one 5y ago
  This isn't unique to event sourcing, I definitely took the opportunity to soapbox.
hcarvalhoalves 5y ago
Event sourcing is reifying a state machine transitions explicitly (in data). So the properties will depend on how the state machine is modeled. If you don’t have strong ordering guarantees, it also means your state machine has to accept transitions from/to any state (eg comutative). Otherwise if ordering is important, your event log must have some ordering built-in (exactly once semantics) or you implement re-ordering and replay as past events arrive (a branching model).
Floegipoky 5y ago
What is driving the adoption of an event sourced model other than a need for properties/guarantees that a reasonably normalized relational model can't provide? If the engineering team isn't talking about tradeoffs and the formal properties of the system, that seems like an indictment of the local engineering culture rather than anything to do with event sourcing.

justusw 5y ago

Is Event Sourcing compatible with GDPR?

ddek 5y ago
The issue with GDPR is right to erasure, which is tricky in conventional systems. But in ES systems, naively implemented, it is not possible. If you are writing PII in plain text, an immutable log won't let you remove it. OTOH I have two options: store PII in a separate database, and just delete or nullify when requested; or use per-user encryption, and delete the key.
- EdwardDiego 5y ago
  Yeah, for GDPR compliance when you're using Kafka, the key is to use, well keys, and log compaction. Then you can retrieve PII for a GDPR if needed. Or, remove it entirely by submitting a "tombstone" record with key -> null, then log compaction removes it. Which basically means, don't mix PII into transnational data.
- lucian1900 5y ago
  The GDPR concept is "personal data", partly since it doesn't just refer to ways to identify a person. PII is a similar but different US concept.
dairyleia 5y ago
An Event-sourced system can be compliant with GDPR, with crypto shredding: https://www.eventstore.com/blog/protecting-sensitive-data-in-event-sourced-systems-with-crypto-shredding-1 https://www.eventstore.com/blog/protecting-sensitive-data-in...

bruth 5y ago

The title here is being overlooked, its not _terrible_, its _hard_. Also this quote from the article: > The bulk of these probably fall under "he obviously didn't understand X," or "you should never do Y!" in which case you would be absolutely right. The point of this is that I didn't understand the drawbacks or pain points until I'd gotten past the "toy" stage. Most of what they discussed was indeed not great, but that was due to lack of understanding. I don't think anyone who does or promotes ES says it's _easy_. However, I would argue it _conceptually_ simple to understand (herein lies the fallacy people make simple != easy). Like any pattern or tech that you are new too, it's going to be hard to build and operate in production (beyond the toy). Everyone takes for granted the decades we've had great tooling and operational patterns for state-oriented databases (mostly RDBMS'). And to be clear, event sourcing is "persistence strategy". What is mostly shown/discussed in the article is "event streaming" along with CQRS. They concepts are largely conflated. Once you understand, you won't be bamboozled. Not to throw Confluent under the bus, but they have unfortunately written or promoted many articles for "event sourcing" use cases and this has confused everyone. You _can't_ do event sourcing with Kafka. It doesn't support optimistic concurrency control at the key level (where as something like EventStoreDB does natively or even NATS now with JetStream). Kafka if fantastic for what it is designed to do.. event streaming.

geoduck14 5y ago
I've never heard of the term Event Sourcing. I AM familiar with Event Streaming. Some of what this article talks about surprised me - and I think it is because I don't fully understand the difference between Event Sourcing and Event Streaming. Is this article a fine summary?: https://developer.confluent.io/learn-kafka/event-sourcing/event-sourcing-vs-event-streaming/ https://developer.confluent.io/learn-kafka/event-sourcing/ev...
- bruth 5y ago
  I have not watched that clip, but as I said above, Confluent isn't a good resource for defining this term since Kafka cannot be used to do ES. I would suggest an article like: https://domaincentric.net/blog/eventstoredb-vs-kafka https://domaincentric.net/blog/eventstoredb-vs-kafka Also refer to the list of great resources @oskar_dudycz posted in another comment: https://github.com/oskardudycz/EventSourcing.NetCore#1319-this-is-not-event-sourcing-but-event-streaming https://github.com/oskardudycz/EventSourcing.NetCore#1319-th...

EdwardDiego 5y ago

Most common issue I've ever encountered with event sourcing is that devs submit events that when replayed, lead to invalid states. And if you're using an immutable log (Kafka/Pulsar/Liftbridge etc.) fixing those mistakes is hard. And given you're basically recording interim states of a state machine, how do you verify that event Z being submitted after event Y is a valid state transition? Validating a record against a schema is trivial. But validating three consecutive events against the state machine? I'm not aware of a good way to do that cheaply. I've done it before using apps that sat in the data pipeline, and maintained state to reject invalid transitions, but that can end up being GiB of state, then you have to persist that across restarts or horizontal scaling. It's all doable, but it costs. Is the event streaming paradigm actually worth the cost? I haven't been convinced yet. But, I'm a big fan of streaming DB change events for auditing purposes - but those are coming from software that already enforces valid state changes. So maybe the answer is - every event has to be submitted via Postgres.

Lhiw 5y ago
> Most common issue I've ever encountered with event sourcing is that devs submit events that when replayed, lead to invalid states. This doesn't sound like event sourcing. It's likely your events aren't events or someone changed what your event handlers do at a later date instead of releasing a new version of the event.
randolorian 5y ago
Seems like a fundamental misunderstanding to me. Commands get validated via your aggregate, not events. Events are not allowed to be rejected. Ever. Because they already happened.
- EdwardDiego 5y ago
  Thinking that event sourcing and CQRS are synonyms is a fundamental misunderstanding also. Anyway, pedantry aside, my point remains thus - it's hard to prevent invalid state transitions being submitted to an event sourcing datastore, and when devs can submit events directly, then these invalid transitions will occur.

Liron 5y ago

Ok but what if the event log was a decentralized database with money built in, would that help?

Liron 5y ago

I think there's a middle option between (1) building your application up from a low-level stream-of-events data model and (2) having your application full of logic to snap from one high-level state to another without a good data-layer audit trail: I'm talking about indexes. Yes, the index part of your database is amazing, because it constantly snaps itself into various states, and you know exactly why: because it needs to maintain itself in the same deterministic relationship with your models. I think that if databases would support more complex and composable indexes, like providing the ability to index join queries, then application programmers would be more nimble and bug-free working with their data. I've written more about this here: https://lironshapira.medium.com/data-denormalization-is-broken-7b697352f405 https://lironshapira.medium.com/data-denormalization-is-brok...

ilitirit 5y ago

One thing that I've learned from Event Sourcing is that it makes bad engineering teams and processes worse.

oskar_dudycz 5y ago

As this article pops out again, I'd like to the point that although it may have some valid points, those points are not about Event Sourcing. What's expressed in the article is the Event Streaming or Event-Driven approach. So when events are not the source of truth etc. All of the event stores that I know supports strong consistency on appends, optimistic concurrency. Many guarantee global ordering. Some help in idempotency. All of that helps to reduce those issues. In Event Sourcing, events are the state. So the flow looks like that: 1. You get the events from the stream (that represents all the facts that happened for the entity), 2. You apply it one by one in the order of appearance to build the current state, 3. You run the business logic and, as a result, create the event, 4. You append a new event. You don't need to cache the write model state anywhere, as the state is in events. Event Sourcing has its issues and troubles, but the stuff described in the article apply to tools like Kafka, etc. They're not tools for Event Sourcing but Event Streaming. They're designed to move things from one place to another, not to be used as durable databases. I've made a list of articles like that: https://github.com/oskardudycz/EventSourcing.NetCore#this-is-not-event-sourcing https://github.com/oskardudycz/EventSourcing.NetCore#this-is... I'll repeat it again: this is not Event Sourcing.

kreetx 5y ago
BTW, does there happen to be a discord channel for this topic? ;)
- oskar_dudycz 5y ago
  There is a Slack channel: https://join.slack.com/t/ddd-cqrs-es/shared_invite/zt-m3vf3alt-S3L~YUoIV88wekj6wSNrUQ https://join.slack.com/t/ddd-cqrs-es/shared_invite/zt-m3vf3a...
nhumrich 5y ago
I re-read the article after this comment, and I would have to disagree. The article does never attempt to explain "what" recent sourcing means to them, so it's hard to know for sure. However, they do mention populating state from an event log which contains meaningless events, so I have to assume they _are_ talking about event sourcing. What leads you to believe they are taking about just streaming, and not sourcing?

DrBazza 5y ago

I've worked on a few event sourcing systems, and the only ones that have been truly successful are the ones where the system was written from scratch. Trying to retrofit systems never seemed to quite work as expected. Also, the "from scratch" systems, seemed conceptually easier to not only understand, but extend, because you were forced to write code in an idiomatic and consistent way. Everything was an event (albeit, a "command" and an "event" confirming 'execution' of the "command").

bob1029 5y ago
I am currently looking at a new iteration of our product that uses an event log as the principal data store. I agree that if we start from "zero" and don't even provide the option to build things the wrong way it would go smoothly. "everything is an event" is very powerful if its not just some add-on policy. For us, the entire working set would be able to live in memory. Larger objects would be left as pointers to the actual data in the log. Snapshots of working set would be taken on a daily basis in hopes of keeping recovery under ~5 minutes during service restarts or updates.
- DrBazza 5y ago
  > Snapshots of working set would be taken on a daily basis in hopes of keeping recovery under ~5 minutes during service restarts or updates. Exactly this. We took hourly snapshots (for a trading system), and could restart the system in a second or so. Another nice side effect of an event driven system is that you start to think about what other data you can put in there. And then you realise you don't necessarily need a database any more, which makes the architecture of the system simpler. There are so many benefits working with a data-driven system. Once data is in the log/ledger/whatever you call it, it's immutable, which leads to parallelism if you want it. Ensuring the system is idempotent also means you get (easier) failover as well.
  - bob1029 5y ago
    > And then you realise you don't necessarily need a database any more This is precisely where we want to be. We are already exceptionally minimal when it comes to database usage - SQLite is all we use today. If we are in a place where 100% of the business facts go to an immutable/append-only log, we can handle the entire persistence mechanism with a simple filestream write command. Not having to worry about mutating prior data solves a large portion of database engine problems automagically.

xg15 5y ago

> The idea of a keeping a central log against which multiple services can subscribe and publish is insane > In effect, the raw event stream subscription setup kills the ability to locally reason about the boundaries of a service. Under "normal" development flows, you operate within the safe, cozy little walls which make up your service. I wonder if this isn't more a case of the "cozy little walls" turning out to be a lie. If your business requirements are that this user action has to trigger those half-dozen barely related background tasks, then your design will always be messy and complex, no matter if you use service calls, event queues, polling or whatever else. In such a situation, I think a better question would be which design lets you most easily reason about the entire system - e.g., what is the dependency structure, how is the causal chain organized, etc. Of course there are also "secondary" concerns, such as: Which design lets you make changes to the system easily, can you still easily develop and test things in isolation that can be isolated, etc. I think services and events actually have some upsides here (but I still may be in the "seduced" phase...) I think this calls more for a better API/dependency management (and treating events as full-fledged APIs!) and a good tracing solution which lets you follow an action through different services (and event queues). > You're free to make choices about implementation and storage and then, when you're ready, deal with how those things get exposed to the outside world. It's one of the core benefits of "services". However, when people are reaching into your data store and reading your events directly, that 'black box' property goes out the window Again, I would absolutely treat events just as much as a public API as REST. Only because the causal realtionship is inversed doesn't mean you can throw away schemas, versioning, dependency tracking etc.

adamauckland 5y ago

Event Sourcing is absolutely brilliant if you want to build an offline first/distributed system which will become eventually consistent. A good example would be a tree inspection application where jobs can be pushed out to mobile inspectors who might not have phone signal but can still collect the data. Once synchronised, views are simply additive. More data can be added to cases simply by adding events. I would absolutely not use ES for a hotel booking system because that data needs to be as accurate as possible in the moment and we've had record locking techniques for over 40 years. Seems a lot of people want to try ES without really considering whether it maps to the real-world domain of the problem they're trying to solve.

picardo 5y ago
> A good example would be a tree inspection application where jobs can be pushed out to mobile inspectors who might not have phone signal but can still collect the data. Once synchronised, views are simply additive. More data can be added to cases simply by adding events. This isn't a great use case for Event Souring either. An inspector must be in front of the tree to inspect it, right? And they can see each other inspecting the tree.... Why would several inspectors inspect the same tree? Assuming they inspect the same tree at different times, you can sync offline data at the row level, and let the last update win, provided that they're entering the same data.
- adamauckland 5y ago
  Where did I say multiple inspectors are inspecting a tree? There might indeed be multiple people dealing with a tree. The tree owner, an inspector, a tree surgeon, the local authorities (if the trees are protected). Synchronising data at row level when offline becomes online is extremely hard to get right and reliable. With the Event Sourcing model, every new piece of data gets thrown into the bucket , with each new piece of data the view becomes bigger.
  - picardo 5y ago
    > Synchronising data at row level when offline becomes online is extremely hard to get right and reliable. Why? Are the users in different time zones? If the time on their devices are synchronized, why is it hard to get right? I'm not trying to pick an argument with you, but trying to figure out if you've considered all the possible alternatives.

invisible 5y ago

Adding something like change-data-capture to a system and using that for some of the purposes that event sourcing touts seems like it goes a lot further and doesn't require making a complex, hard-to-understand system. Having a CDC system to publish events and record those can give the biggest benefits and sit alongside a much simpler system. A downside is that the real-time feature goes away.

oftenwrong 5y ago

This matches closely with my experience working at a company that was mostly built on event-sourcing. I would add a few things, some of which are touched upon in the article: - The general problem with event-sourcing (at small scale) is that it forces you to have to think about and handle many things that can mostly be ignored with a more typical persistence approach. This overhead of complexity was one of the factors that (IMO) killed a company I worked for. Event-sourcing feels relatively "raw" in this way. However, this also provided a valuable learning experience as a developer. - I'm curious about better ways to do event-sourcing. For example, using Datomic is apparently like a well-thought-out, pre-made event-sourcing system: https://vvvvalvalval.github.io/posts/2018-11-12-datomic-event-sourcing-without-the-hassle.html https://vvvvalvalval.github.io/posts/2018-11-12-datomic-even... - If you intend to embark on an event-sourcing journey, read Versioning in an Event- Sourced System https://leanpub.com/esversioning/read https://leanpub.com/esversioning/read , even if only to better understand the complexity of this one aspect. I agree with the author of this submission that it is hard to fully grasp the complexities of event-sourcing without trying it, and hitting many road bumps along the way, since it's not an especially well-trodden path. - Being able to redefine the interpretation of an event as the system evolves feels like a superpower at times. Similarly, being able to bring a new part of the system online, and have it based on a rich history of everything ever done in the system, is very cool. One of the major benefits of event-sourcing is how adaptable it is in systems that change over time. - Beware CQRS and eventual consistency. This adds a lot of complexity and is not a prerequisite for event-sourcing. It's also telling that ordinarily-simple problems to solve, like preventing duplicate username registrations, are hand-waved away by CQRS proponents: https://web.archive.org/web/20191101010824/http://codebetter.com/gregyoung/2010/08/12/eventual-consistency-and-set-validation/ https://web.archive.org/web/20191101010824/http://codebetter... . You will face this class of problems, and you may not be able to hand-wave it away so easily.

valw 5y ago

I submit that Event Sourcing (or at least something very close to it) can be easy, once you've removed some technology locks. I've seen it happen with Datomic (https://vvvvalvalval.github.io/posts/2018-11-12-datomic-event-sourcing-without-the-hassle.html https://vvvvalvalval.github.io/posts/2018-11-12-datomic-even...), but it can probably happen similarly with other tech, such as XTDB (https://xtdb.com/ https://xtdb.com/).

ryanmarsh 5y ago

This is a balanced article and I appreciate the author’s treatment of the challenges. Part of my work is helping enterprise shops with n-tier architectures decouple services to allow for more resilience end to end. Everything the author states has crept up, some of the issues I consider the “nominal cost of resilience and independence”. Other issues are easily mitigated, such as the yak shaving required to get off the ground. Serverless solves the infra/config challenges. If an enterprise shop isn’t open to serverless I have no interest in discussing event sourcing or CQRS with them. It’s too much work otherwise. For shops that are being crushed by fluctuations in concurrency, challenges with the uptime and p50 of partner teams, don’t have much of a choice but to go event sourced, optimistic, and eventually consistent. I’m certain a great enterprise IT shop with great leadership could mitigate the challenges of service growth and instability with some clever optimizations of their current stacks but unfortunately I have to meet a shop that is uniformly competent. Event sourcing helps with the competency gap problem. If every team’s data is on the outside my team can work around your lead time for changes or general ops incompetence. If inventory is down, fuck you I’m still taking orders. The extra work of maintaining a projection of your inventory events makes good business sense because time kills all deals.

acjohnson55 5y ago

This article is great. I architected an ES system once. It was absolutely the right call, but we had to work through nearly every one of those problems. Regardless of whether you do it well or not, engineers will find it cumbersome to work with and think that it's overengineered.

swyx 5y ago

I work at Temporal[1], where we solve a bunch of these problems for you by cleanly abstracting the critical event sourcing bits under our SDKs. I think everyone who tries event sourcing eventually rolls their own framework or takes one of the shelf, and Temporal is the most battle tested one I've ever found. The reason I posted this was that I found it helpful in prepping for a workshop I was giving today - recording is not up yet but slides are here https://docs.google.com/presentation/d/1mAPYXNmRdDKepXylB91F2uqFyHm8rG12K2PMRHxoUfM/edit?usp=sharing https://docs.google.com/presentation/d/1mAPYXNmRdDKepXylB91F... 1: https://temporal.io https://temporal.io

pm24601 5y ago

Also showed up on hacker news in 2019 https://news.ycombinator.com/item?id=19072850 https://news.ycombinator.com/item?id=19072850

mixmastamyk 5y ago

What the hell is event sourcing? I read the first half of the article and gave up. Sounded like one of those distributed log things like kafka or maybe a message broker like rabbit? Yes, could google, but now am discouraged. If you're going to write one of these, at least link to the wikipedia page: https://en.wikipedia.org/wiki/Domain-driven_design#Event_sourcing https://en.wikipedia.org/wiki/Domain-driven_design#Event_sou...

bartread 5y ago

I worked on an event sourced system once, many years ago. I'm not saying I hate the separation of write and read models, and all the challenges that brings (heck, the platform I'm working on now does that, and it's not a big deal), but it was massively bloated and overly complex for a system that would collect some data from users, submit it to a panel of third party services, and display results back to those users. The reasons for going this route were complex, and not entirely clear to me, although I can pick on two: firstly, they'd run into performance and scalability issues with their existing CRUD-ish SQL Server-backed implementation and, secondly, Thoughtworks told them it would be a good idea. I don't want to handwave this away but I've encountered enough insoluble SQL Server performance issues over the years to be confident those issues could have been solved much more simply (and the same is likely true for Oracle, Postgres, et al). I'm not suggesting that intractable performance issues don't exist on relational platforms, just that most times people think they've encountered one, they're mistaken. Would I use event sourcing again? Maybe. But I'd need a really strong use case for it, and I'd certainly kick the tyres on any technical organisation that was using it before I joined.