4 ms·

I think more and more things will be developed like this instead of just writing CRUD apps. It kind of makes sense since storage is super cheap, lambdas and oth

by 0xCMP 8y ago

I think more and more things will be developed like this instead of just writing CRUD apps. It kind of makes sense since storage is super cheap, lambdas and other event-driven systems are getting popular, and it fits really nicely with start-ups being able to pivot existing data to a new schema/database/analytics. I've read before this is not recommended by people who did it (besides people who are consulting firms or promote Kafka). Is there a reason for this anyone can chime in with?

vorpalhex 8y ago
Event sourcing and CQRS as a whole leads to horrific complexity quickly, especially if you violate certain constraints. How do you synchronize multiple events? How do you handle partial system outages? What is the retry strategy for failed events and how do we handle inconsistent data? Then comes the fact that you've more then doubled your data size since you need your source of truth and a copy on each read view - that gets expensive quickly. I agree with Fowler - CQRS/ES is probably too complicated and you should avoid it unless you have a well described bounded domain that fits the model.
- dfee 8y ago
  I discovered the promises of CQRS/ES through Martin Fowler. I didn’t realize he had discounted it to the extent you’ve said, but I’d like to read his thoughts. Do you have a link to the article, or is it in a book?
  - pdimitar 8y ago
    They are probably referring to this: https://www.martinfowler.com/bliki/CQRS.html https://www.martinfowler.com/bliki/CQRS.html
- joevandyk 8y ago
  If you can store events in a sql database and use transactions and use idempotent APIs, a lot of those issues go away.
  - vorpalhex 8y ago
    If you can do those things, you don't need cqrs to begin with.
- scruple 8y ago
  > Event sourcing and CQRS as a whole leads to horrific complexity quickly, especially if you violate certain constraints. This has been my lived experience with it, as well. I find that, in retrospect, the complexity was really not worth it. And now, with regulation like GDPR, how do you handle things like the right to be forgotten in an (and especially if it's a legacy app) ES/CQRS architecture? What I've personally discovered as the approach we implemented at work is horrifically complex, to borrow your words.
- yen223 8y ago
  "How do you synchronize multiple events? How do you handle partial system outages? What is the retry strategy for failed events and how do we handle inconsistent data?" Funny thing is that the traditional stateful model handles these scenarios horrendously as well. Probably more so, since you have no history to reconcile with.
- codebje 8y ago
  > How do you synchronize multiple events? How do you handle partial system outages? What is the retry strategy for failed events and how do we handle inconsistent data? Make an event sourcing system single threaded, and the problems of synchronisation don't exist. Make a "traditional" RDBM system distributed, and the problems of synchronisation exist. If you weren't building a distributed (or parallelised) system to start with, why introduce it with event sourcing? If you were, how were you going to solve all those problems with a traditional RDBMS approach? The solutions are more or less the same. Perhaps "retry strategy for failed events" is new - how would you have handled a replicant failing in a distributed RDBMS application? The data size issue in my experience pales in comparison to the problem of having the business ask questions you can't answer because you threw away data. How high is your transactional rate, anyway? Views shouldn't be a data size concern, as they should only contain the pertinent data. In many cases you should be able to hold views completely transiently in memory. It's relatively rare that a view is expensive to compute from an ordered history. > I agree with Fowler - CQRS/ES is probably too complicated and you should avoid it unless you have a well described bounded domain that fits the model. Given the model fits any domain in which things happen, this might not be the best way to determine if ES is appropriate - it's probably more along the lines of how important history is, how important flexibility for future use of data is, and how important time to market is. If we give up security to get a product out sooner, we would pretty quickly also give up a total history of the system, too. There's also more than one way to skin this cat - you can use transaction scripting and log transactions; you can use database table level auditing; you can have a high level business intent audit log. Probably others. They all have complexities and drawbacks. I don't personally find CQRS/ES particularly complicated, having worked with it a few times. I don't apply to every project, the same way I don't apply any of the other history mechanisms to every project, but I'll reach for it over those other mechanisms, because I prefer the costs of ES over the costs of those mechanisms.
- jen20 8y ago
  In other words, you should avoid using a model aimed at simplifying a domain model if you don't have one? Sounds fair... The canonical description of the domain model pattern (from Fowler, also) calls for it be _considered_ in a case where there are "complex and ever-changing business rules". He goes on say that "if all you have is some sums and null checks, a different transaction processing pattern is more appropriate". I have seen a lot of event sourced systems, and the vast majority which fail fall into the "not null checks and trivial sums" case rather than the former.
superzadeh 8y ago
I wokred a lot with CQRS/ES before and looked into it when building our tech stack of a new startup. My takeaway is that the complexity it brings would simply get in the way for a new system that don't have a lot of complexity. You will simply get things faster without it, and can always introduce it later to progressively phase out components with lots of debt.

by 0xCMP 8y ago

vorpalhex 8y ago

Event sourcing and CQRS as a whole leads to horrific complexity quickly, especially if you violate certain constraints. How do you synchronize multiple events? How do you handle partial system outages? What is the retry strategy for failed events and how do we handle inconsistent data? Then comes the fact that you've more then doubled your data size since you need your source of truth and a copy on each read view - that gets expensive quickly. I agree with Fowler - CQRS/ES is probably too complicated and you should avoid it unless you have a well described bounded domain that fits the model.

dfee 8y ago
I discovered the promises of CQRS/ES through Martin Fowler. I didn’t realize he had discounted it to the extent you’ve said, but I’d like to read his thoughts. Do you have a link to the article, or is it in a book?
- pdimitar 8y ago
  They are probably referring to this: https://www.martinfowler.com/bliki/CQRS.html https://www.martinfowler.com/bliki/CQRS.html
joevandyk 8y ago
If you can store events in a sql database and use transactions and use idempotent APIs, a lot of those issues go away.
- vorpalhex 8y ago
  If you can do those things, you don't need cqrs to begin with.
scruple 8y ago
> Event sourcing and CQRS as a whole leads to horrific complexity quickly, especially if you violate certain constraints. This has been my lived experience with it, as well. I find that, in retrospect, the complexity was really not worth it. And now, with regulation like GDPR, how do you handle things like the right to be forgotten in an (and especially if it's a legacy app) ES/CQRS architecture? What I've personally discovered as the approach we implemented at work is horrifically complex, to borrow your words.
yen223 8y ago
"How do you synchronize multiple events? How do you handle partial system outages? What is the retry strategy for failed events and how do we handle inconsistent data?" Funny thing is that the traditional stateful model handles these scenarios horrendously as well. Probably more so, since you have no history to reconcile with.
codebje 8y ago
> How do you synchronize multiple events? How do you handle partial system outages? What is the retry strategy for failed events and how do we handle inconsistent data? Make an event sourcing system single threaded, and the problems of synchronisation don't exist. Make a "traditional" RDBM system distributed, and the problems of synchronisation exist. If you weren't building a distributed (or parallelised) system to start with, why introduce it with event sourcing? If you were, how were you going to solve all those problems with a traditional RDBMS approach? The solutions are more or less the same. Perhaps "retry strategy for failed events" is new - how would you have handled a replicant failing in a distributed RDBMS application? The data size issue in my experience pales in comparison to the problem of having the business ask questions you can't answer because you threw away data. How high is your transactional rate, anyway? Views shouldn't be a data size concern, as they should only contain the pertinent data. In many cases you should be able to hold views completely transiently in memory. It's relatively rare that a view is expensive to compute from an ordered history. > I agree with Fowler - CQRS/ES is probably too complicated and you should avoid it unless you have a well described bounded domain that fits the model. Given the model fits any domain in which things happen, this might not be the best way to determine if ES is appropriate - it's probably more along the lines of how important history is, how important flexibility for future use of data is, and how important time to market is. If we give up security to get a product out sooner, we would pretty quickly also give up a total history of the system, too. There's also more than one way to skin this cat - you can use transaction scripting and log transactions; you can use database table level auditing; you can have a high level business intent audit log. Probably others. They all have complexities and drawbacks. I don't personally find CQRS/ES particularly complicated, having worked with it a few times. I don't apply to every project, the same way I don't apply any of the other history mechanisms to every project, but I'll reach for it over those other mechanisms, because I prefer the costs of ES over the costs of those mechanisms.
jen20 8y ago
In other words, you should avoid using a model aimed at simplifying a domain model if you don't have one? Sounds fair... The canonical description of the domain model pattern (from Fowler, also) calls for it be _considered_ in a case where there are "complex and ever-changing business rules". He goes on say that "if all you have is some sums and null checks, a different transaction processing pattern is more appropriate". I have seen a lot of event sourced systems, and the vast majority which fail fall into the "not null checks and trivial sums" case rather than the former.

superzadeh 8y ago

I wokred a lot with CQRS/ES before and looked into it when building our tech stack of a new startup. My takeaway is that the complexity it brings would simply get in the way for a new system that don't have a lot of complexity. You will simply get things faster without it, and can always introduce it later to progressively phase out components with lots of debt.