13 ms·

I have worked on, or cleaned up, 4 different CQRS/ES projects. They have all failed. Each time the people leading the project and championing the architecture

• devbug 10y ago
  Do you mind detailing your experience with Dataomic? We're looking at ways to store 6-ary tuples (quadstore + temporality of existence and temporality of observation) of facts and build indices on top of them. I'm hesitant to move to (relatively) obscure data store without a really good idea of where that puts us.
  • eric_b 10y ago
    We didn't get terribly far down the road with Datomic. Management/administration of the DB was not for the faint of heart. (At the time the docs were simply terrible, maybe they are better now?) We would see data corruption/loss as well. We weren't doing anything terribly complicated with it, and data loss doing the simplest things was unacceptable. (It's entirely likely we were the cause of the data loss somehow, but we followed the guides and did nothing crazy or weird). So anyways, we just threw it out almost immediately. Conceptually it wasn't really a fit with the team or our application either. The query model and direct reading from the storage medium seemed silly - why not just use underlying storage medium directly for everything? (Besides, now you have to administer both Datomic and Cassandra/Oracle whatever) Not worth it for the obtuse query approach and limited value of "never forgetting" imo :)
    • Skinney 10y ago
      Direct use of the storage medium for everything would negate the advantages Datomic offer. The beautiful part of Datomic is that you can perform an expensive query without having any effect on other peers. Since Datomic also stores datoms in blocks, the n+1 problem is reduced as well. Another cool thing is that for unit-testing, you can just disconnect Datomic from the storage medium, and run everything in memory. Datomic requires writes to be synchronised, which is why you can't go directly through the underlying storage medium for write cases. While I personally would love to use Datomic for just about everything. The fact that I need at least three machines going for the simplest app (1 actual database, the peer and the transactor) and that those machines can't be the cheapest machines (you need enough ram to store datoms in a cache on the client, or things will be slow), Datomic is something I can rarely afford in practice.
• ryanworl 10y ago
  What do you think about the use of technologies like Kafka which enable what is effectively an event sourced architecture without all of the buzzwords? Not everyone uses it that way, obviously, but there is plenty of discussion about it.
  • eric_b 10y ago
    One of the projects did use Kafka as the "event log". There were stability issues with the version of Zookeeper that was used. From a writer/reader perspective Kafka was sufficiently performant when it was up. (The Zookeeper issue was eventually fixed as I recall, but by then the damage was done in terms of political capital spent and lost.) The big issues didn't really have that much to do with the persistent store of events. The bigger issue was the fact that as new features get added to your application, your event payloads change. Well, in order for projection (particularly re-projection in the case of an issue) to work, your code needs to know how to read and process all versions of every event. Of course, there are techniques like snapshotting to give you point in time "good states" so you can eventually deprecate some of these events, but thinking about it is challenging. Additionally, most folks argue that the event store should be immutable. This is great until some kind of bad event gets in there. Now your code needs to know how read, and discard, this bad event forever (or until a snapshotted point in time). Finally, projection is not the panacea the evangelists will have you believe. Inevitably there will syncing issues between the event store and the projected database/elasticsearch cluster/mongo instance whatever. And what do you do then? Re-project! But that is not easy :)
    • slashdotdash 10y ago
      For migration of immutable events, there's a good research paper[1] that outlines five strategies available: multiple versions; upcasting; lazy transformation; in-place transformation; copy and transformation. The last approach even allows you to rewrite events into an entirely new store. [1] The Dark Side of Event Sourcing: Managing Data Conversion http://files.movereem.nl/2017saner-eventsourcing.pdf http://files.movereem.nl/2017saner-eventsourcing.pdf
    • spuz 10y ago
• baconforce 10y ago
  Would love to hear about your experience and why CQRS/ES failed those projects, and what type of technical complexities it introduced.
  • eric_b 10y ago
    I detailed some of the problems I saw in another thread, but one other thing that gets challenging is the CQRS side (if you choose to keep everything async). For example, say you fire an "address change" event. That gets sent to the event store for eventual projection in to a medium you can actually query from (realtime querying of the event store itself is the road to very bad places, I promise). So now your event is sent along, but how do you know when it has been processed/projected? What if there was a conflict with another address change event from somewhere else? How does that bubble back up? The typical solution is to pass back some kind of "receipt token" so you can poll to see when your event is processed, then do your read from the projected database, or whatever. Of course, this can be made to work, but once you start talking edge cases and the need to support standard UX paradigms, polling for every update and handling error scenarios in this way becomes painful.
    • brightball 10y ago
      Based on his article it looks like he's using subscriptions internally rather than polling. That's a fairly natural thing to do across an Elixir application/cluster. In terms of conflict resolution, it seems like you'd have to clearly define a scenario where a conflict was possible. Based on the write-up, the state of an address would be based on the aggregate of the events that wrote to it. That seems like it would always lead to the last change winning. From the write up of the system, I actually can't imagine trying to do this in anything other that Elixir/Erlang. The set of requirements and challenges to pull it off would be really complicated on just about any other platform.
    • UK-AL 10y ago
      The address thing is normally solved by the fact that you organise your commands by things that should only logically change togther. So a conflict messages won't revert irrelevant fields back to their old ones. So your commands should not be UpdateCustomer They should be UpdateCustomerAddress UpdateCustomerEmail Etc For the address, just take the last one. All the business logic I can think makes this ok.
• UK-AL 10y ago
  I've worked on a successful CQRS project. It didn't use es though. Probably one the nicest systems I've ever helped build. We had a very good lead though.
  • jbogard 10y ago
    Oh yeah, us too. We use CQRS on nearly all projects we do these days. But event sourcing, totally different animal.
  • eric_b 10y ago
    I like thinking about applications in terms of read-side and write-side. I'm not super enamored with async CQRS, but I am a huge fan of APIs and data models where the developers realized that how I consume the data is wildly different than how I update it.
  • zdbo 10y ago
    Absolutely. I use CQRS fairly regularly, but I've never used ES on production software and I can't think of a case where it would have been appropriate. It's a serious solution that is not trivial to implement. CQRS is a highly understandable, almost always reasonable technique to follow when you strip away all the other things that people tend to associate it with. This is a pretty decent article on that: https://lostechies.com/jimmybogard/2012/08/22/busting-some-cqrs-myths/ https://lostechies.com/jimmybogard/2012/08/22/busting-some-c...
• slashdotdash 10y ago
  I'm the author of this article. It sounds like you have some valuable, real-world experience with CQRS/ES. I'd love to read more about the difficulties you've faced, and overcome. For migration of immutable events, there's a good research paper[1] that outlines five strategies available: multiple versions; upcasting; lazy transformation; in-place transformation; copy and transformation. The last approach even allows you to rewrite events into an entirely new store. [1] The Dark Side of Event Sourcing: Managing Data Conversion http://files.movereem.nl/2017saner-eventsourcing.pdf http://files.movereem.nl/2017saner-eventsourcing.pdf
  • karmajunkie 10y ago
    Good link! Looking forward to reading that. I've been following your projects on Github for awhile, good work—I don't necessarily agree with all of the design choices but we've built on the eventstore at work and I'm going to be using it on another project in the near future.
    • slashdotdash 10y ago
      Please do feedback your ideas to improve these open-source projects. I'd be interested to find out how you're using CQRS/ES in Elixir.
      • karmajunkie 10y ago
        We've got a channel (#eventsourcing) on the elixir slack—low volume but frequently interesting discussions, please feel free to join us there as well.
  • kiwidrew 10y ago
    Thanks for the link to the research paper, it's a good read. Are you aware of any event sourcing frameworks/datastores that use one (or more) of these strategies to tackle the problem of schema evolution?
    • wmfiv 10y ago
• jbogard 10y ago
  Wow, this succinctly sums up our experience. Fun to develop against, absolute nightmare to support in production. The only place I'd recommend it these days are where the business views their state as an event stream, maybe finance/stocks. Not developer-forced-events like "customer address updated" or "user email changed". Even workflow systems I've dealt with, the business doesn't view their state as an event stream. The state is where it is, how it got there is an interesting footnote.
  • jen20 10y ago
    No shit... if you listen to the actual talks, that is EXACTLY what they recommend. Use the patterns where appropriate, don't use it where it's not appropriate. Also consider _where_ the advice came from - finance, gambling, healthcare and so forth. As for "not production ready" with regards to Event Store specifically, we have had zero reported incidents of data loss which were catastrophic failure of the hardware (or, usually the cloud) it was running on.
    • eric_b 10y ago
      One of the projects I worked on was in finance. The fact that we were going to get "a free audit log" from the architecture made everyone so excited. After the CQRS/ES project failed (I was on cleanup duty) we used a more traditional arch. To handle the audit log we just had a separate table. ("customer" table had "customer_audit" table. Both were written to in transaction. Solved.)
      • deleted 10y ago
        [deleted]
    • jbogard 10y ago
      Not production ready - the system, not the database. We never had a problem with the event data stores. The bespoke systems our CQRS/ES rescuees built around it, absolutely.
• jen20 10y ago
  I would be interested to know why you consider ES "not production ready". There have been zero reported incidents of data loss in stable released versions which were not as a result of catastrophic failure of the hardware on which it was running (though most people's poor understanding of Azure has caused more problems than everything else put together). Furthermore, this should be no surprise given the testing process[1] continually run. [1]: https://geteventstore.com/blog/20130708/testing-event-store-ha/index.html https://geteventstore.com/blog/20130708/testing-event-store-...
  • eric_b 10y ago
    We had no data loss incidents. My concern was around tooling and maintenance of the database. Once you had to start clearing out bad events in dev environments or doing common administrative chores there were many holes. I never did find out how to remove specific problem events. We ended up just purging the store every time something went wrong in lower environments. We had no solution for this problem in prod. How would you fix an event that should not have gotten in to the store? (Wrong contract, bad data etc) I understand it shouldn't happen, but in the real world all kinds of things go wrong. I can't imagine running a massive system on something like event store. Fortunately the projects always failed well before we got in to any kind of production with real users.
    • ehsanu1 10y ago
      Not saying this is great, and I don't have experience with an ES system in production, but... Could you do one of: 1. Manually remove the specific problem event from the store, then re-run all events starting from the previous accumulated state snapshot stored before that event? Then you get the new state snapshot just by re-processing all the events. This of course assumes you have state snapshots, and may not be feasible if this is a common occurrence and there are too many events to process. Or: 2. Create a "reverse" event? This will cancel out the bad change and give you a new, valid state to work from and continue. This is nice since the historical state of the system is still represented, but that may be a disadvantage in some situations.
• inopinatus 10y ago
  > I strongly believe it is the wrong choice for just about every application. None of these are pure as-originally-outlined-by-Fowler CQRS/ES, but I'm willing to suggest they are paradigm equivalent, real world successful examples: 1. Basically all double-entry accounting/book-keeping/core banking systems. 2. Many RDBMSes. Specifically the replayable log structure of transactions. 3. I sell a service that includes two event-sourced data structures mutated by domain-specific commands. They are used for collaborative decision making in sports management. This aspect works very well indeed: the resulting characteristics are intrinsic to ES structures, are (AFAIK) unique in our market, and represent one of our most customer-retaining capabilities.
• modarts 10y ago
  I'm curious to hear what experiences you had with Datomic that led you to the conclusion that it wasn't production ready.
• vendiddy 10y ago
  CQRS/ES requires a different way of thinking and a different set of best practices. This could make it easy to shoot yourself in the foot. That being said: - Dealing with failures can be better than traditional systems if done properly. For example, we have services where, if they fail, won't bring down the entire system. However, this does require you to be more explicit on how you handle errors. - I have found debugging to be easier. When an error occurs, we can trace it back to the exact command and the events it generated. This allows us to see 1. the exact state of the system at the time the error-producing command was generated and 2. the exact command that was executed. From this we can easily reproduce the error. I have covered "versioning events" in my other comment. Please be more specific about "projection, reporting, maintenance, administration". What exactly were the challenges there? I understand that ES is not a silver bullet but I would like others to have a clear understanding of the tradeoffs to traditional systems.
• sbellware 10y ago
  I've had similar, unfortunate gigs. Even where there are smart, capable, technically-adept folks in-place, it in no way means that they have full command of the implications and the ways and means of event sourcing. In my experience, event sourcing is useful in most projects - even, contrary to an earlier comment, for the user and user profile concerns. I've worked with EventStore. I've wanted to smash EventStore out of existence on a number of occasions. On some of those occasions, I was at fault. And on others, I found the user experience (as an implementer) misleading, ambiguous, and ultimately costly. I'm quite frank with James and Greg about what I think EventStore should be achieving, and have been quite public about it in the past, so I won't rehash that here. I guess I haven't seen the ebb and flow of excitement, though (since 2007-ish when I started crossing paths with Greg and Udi). I do see a steady rise in awareness of it since then, and I see an increasing number of successful projects. There's also more people helping other people and shops with implementation guidance and safety. And there's bound to be more failures - just as a matter purely of numbers - just as there are with any platform or tool where the learning was underestimated, or the grasp of the whole was overestimated. Yep, you can fail with event sourcing. You can fail with Rails. It will take you longer to fail with Rails, though. And for an organization that isn't into the learning as a matter of course, failing over a longer term can be an important and empowering strategy. I have two sets of customers: those whom I help slow the looming failure of their Rails projects, and those whom I get started on event sourcing. It's not always a good cultural fit. But I have yet to see a domain that's worth the expenditure of a software development team that is somehow naturally inappropriate for event sourcing.