8 ms·
CQRS: Command Query Responsibility Segregation (2011)
- maxekman 6y agoThis has been discussed here many times, but I still thinks it’s worth being reminded about. It’s a very universal pattern that should get more cred for being used on its own without event sourcing.
- kylecordes 6y agoAt work (consulting/development shop), we went far down this path a few years ago. Conclusions: 1. CQRS+ES works great, if you implement it carefully, which requires discipline. We ended up creating our own framework (ugh) to make it easy to go down the right path. 2. There are trade-offs, of course. 3. You are only one smart agile thinker-in-a-pinch away from stepping away from the discipline and creating really tricky complexity and far more negative trade-offs. 4. There is very little market demand for this approach among the typical enterprise business solutions we work on; though as I understand there are some sharp peaks in certain areas of financial services. (The odd thing is... we have financial service customers, just not in the particular domains where there is demand for CQRS.)
- k__ 6y agoWhat were the biggest issues with it?
- born_a_skeptic 6y agoWhat aspect of Financial Services? Trading?
- dsies 6y agoIn my experience, lots of banking, some trading. In both cases, it is used for history, audit and precision. The idea is that for banking, it is not enough to just get the _current_ state - the more important thing is how someone _reached_ that state. Adding history to transactions is not new - so rather than bolting on a history/audit mechanism, you knock out both - a higher resilience, distributed system + built-in audit/history mechanism.
- crispyambulance 6y ago> You are only one smart agile thinker-in-a-pinch away from stepping away from the discipline and creating really tricky complexity and far more negative trade-offs. I love this sentence. That seems to be a common pitfall, regardless of framework or pattern, which ironically puts a hero aura around the person who did it.
- fmax30 6y agoI agree with most of your points. We also went down this path (CQRS + ES) a few years ago at my old company. Implementing it was fun, but it did add quite some complexity. Granted, it was very scalable. Interestingly enough we used it for an enterprise messaging app, although I left for other things before it was released. I wonder how that codes looks today.
- DougBTX 6y agoFrom the end of the article: > Despite these benefits, you should be very cautious about using CQRS. Many information systems fit well with the notion of an information base that is updated in the same way that it's read, adding CQRS to such a system can add significant complexity. I've certainly seen cases where it's made a significant drag on productivity, adding an unwarranted amount of risk to the project, even in the hands of a capable team. So while CQRS is a pattern that's good to have in the toolbox, beware that it is difficult to use well and you can easily chop off important bits if you mishandle it.
- joshsyn 6y agoImplemented CQRS+ES design pattern recently in F# and postgres. The query side kept changing due to business needs, while the command side remained the same. In our case, queries state aka projection were asynchronous which made it very flexible and fast to work with. Projection could live anywhere, within the web server, redis or just database.
- superice 6y agoI like how GraphQL essentially popularized the advantages of a CQRS (without Event Sourcing!) approach for your public API by explicitly giving you different DTOs for queries and mutations. It's a pretty neat way of enforcing this while not being a super heavy handed framework like for instance Axon + Spring Boot tends to be.
- ralusek 6y agoREST did the same with POST/PUT/DELETE vs GET. I mean, technically HTTP did, before that.
- simiones 6y agoNo, REST puts the Resource at the forefront, and normally you use different verbs on the same resource: you PUT/POST to create a new resource, you can then GET that resource to read it, you PUT/PATCH to modify it, you DELETE a resource to delete it. CQRS usually means that Commands (modifying the system) are handled differently than Queries (checking the current state of the system). Of course, you can absolutely create a REST API where there is one set of resources that you can modify but not read (Command resources), and one set of resources you can read but not modify (Query resources). But that would definitely not fit the general idea of what people expect from a REST API. Also, such an API would be relatively hard to make HATEOAS-compliant, for those that care about "ultimate"/"true" REST.
- ralusek 6y agoNo, I wasn't saying that REST was the same as CQRS. I was replying to the comment that made the case that GraphQL was similar to CQRS because it separated reads from writes by having queries and mutations. I'm saying that REST also separates reads from writes in almost identical capacity. Neither of them are comparable to CQRS.
- simiones 6y agoThe parent claimed that GraphQL has different DTOs for reads and writes. That could make it CQRS compatible, couldn't it? I don't know if it would be any more idiomatic for GraphQL than it would be for REST.
- Sodman 6y agoThe biggest drawback I've seen to CQRS - particularly in an event driven architecture - has been increased complexity, usually in exchange for increased scalability. I'm a big fan of "Do the simplest thing that could possibly work" so I'm mostly opposed to CQRS except for some niche cases. A CRUD design for an app can scale well enough for most businesses to get into very decent revenue numbers. When you start bumping up against limitations, then you can start to refactor. Until then, the simplicity of a CRUD app can let you prototype and ship faster, as well as significantly boost new dev onboarding time.
- whalesalad 6y agoWell... you’ve just described the law of nothing in life is free. Of course there are trade offs with any architecture choice.
- AndrewSChapman 6y agoAgree 100%. There are probably better alternatives if you're needing to improve scalability.
- kemiller2002 6y agoIt is amazing how much a well tuned and properly functioning database can handle. At my last job, this was a huge hurdle I was tasked with handling. They're CQRSish design slowed so many things down through needless complication or hidden inefficiencies. People always jump to "we need to scale and this will make it faster" when the real answer is often, "how about we focus on some fundamentals and just make our code more efficient"
- barumi 6y ago> The biggest drawback I've seen to CQRS - particularly in an event driven architecture - has been increased complexity, usually in exchange for increased scalability. I fail to see how CQRS is a factor in increased complexity. After all, CQRS boils down to separating the interfaces for queries/reads/immutable operations and commands/writes/mutable operations. Unless you're bolting on unrelated concepts, like event sourcing, then CQRS is not a significant source of increased complexity in non-trivial applications. Can you shine some light on which aspects led to higher complexity in your use of CQRS?
- alfl 6y agoGreat pattern for financial data / fintechs. Easy to explain to compliance teams. If you go fully event sourced, use gateways: https://martinfowler.com/eaaDev/EventSourcing.html https://martinfowler.com/eaaDev/EventSourcing.html
- revskill 6y agoHow to handle missed/failed events in a distributed transaction ?
- danielovichdk 6y agoThis is not really a CQRS issue. This is more of a system capability around how you wish replaying events should be handled. If you design your system around idempotency you should be able to replay events and get the same result. Not sure it was what you wnted to hear?
- revskill 6y agoI've seen replaying events as patterns on how to handle this case. But in a monothlic architecture, there's no concept as "replaying", isnt' it ? I'd love to see similar handler in a distributed environment.
- morelisp 6y agoHow do you have a distributed transaction in a monolithic architecture?
- dsies 6y agoI might be missing something - what is the issue with being able to handle a replay event stream in a monolithic service? As the previous poster mentioned, if you are are able to handle events idempotently, you should be able to ingest events, regardless of the destination - monolithic or distributed.
- danielovichdk 6y agoCQRS is a great pattern. But like any pattern ist should not be applied system-wide. It's not a fit, imo, as a system architecture. It's more a subsystem design where you apply it to something which has many writes and a eventual consistent read-store, which can of course be built from the write-store. That doesn't really matter. This can aslo easily be done without adding event sourcing, and the two does not per se go hand in hand. The key point being that you sometimes cannot build a view "fast enough", based on data which is constantly entering the system. So the system with in the CQRS design, which handles a query is very important to design based on this. I see it as a pattern that really helps you to perform on the read/query load due to the almost impossibility to make a query perform over a massive amount of commands/writes
- nickbauman 6y agoPrecisely my take too. Maybe we're even right?! My sense is the crux is that you make explicit with CQRS is that you can offload reads to read replicas as long as you're OK with the read replicas being a bit more behind in the writes. Which gives you greater scalability like you say.
- deleted 6y ago[deleted]
- dsies 6y agoYour sense is totally right and to boil it down further - you have to be OK with your system being eventually consistent.
- chrismeller 6y agoNot my first rodeo with CQRS, so I didn’t really learn anything new here, but I was really impressed with how well he laid everything out - particularly the links to other terms that, as a newbie, I felt other resources left me scrambling through a dozen browser tabs to find.
- shekharshan 6y agoTo give a more concrete example, we have a single Angular app. The app has two areas, one where the user can search for records using free text while another where the user can create or update records. Our source of truth is a Postgres database where all updates and inserts are sent using a microservice. The data flows from Postgres to Elasticsearch using SQS in AWS. The Angular app then queries Elasticsearch using a different microservice. So essentially our command domain model is separate from our query domain model.
- 02020202 6y agocqrs is great, but you need to emit changes in your system in consumable way in order for cqrs to work since it is based on reaction to these changes and building the "views" from them. not fun to implement, a lot of code is needed, especially if you are doing event sourcing and not just some simple event logging. but it is VERY powerful design, in conjunction with event sourcing. it essentially removes strict schema from your system and you can have fluid data storage and perform various analytical tasks with past data and not being locked only to current data. very powerful! it is a dream to have but a pain to implement.
- c17r 6y agoBeen looking into CQRS and people's thoughts/examples/implementation are as wide and varied as the stars. The answer straight from the horse's mouth (Greg Young): https://web.archive.org/web/20190211113420/http://codebetter.com/gregyoung/2010/02/16/cqrs-task-based-uis-event-sourcing-agh/ https://web.archive.org/web/20190211113420/http://codebetter... CustomerService void MakeCustomerPreferred(CustomerId) Customer GetCustomer(CustomerId) CustomerSet GetCustomersWithName(Name) CustomerSet GetPreferredCustomers() void ChangeCustomerLocale(CustomerId, NewLocale) void CreateCustomer(Customer) void EditCustomerDetails(CustomerDetails) goes to --------- CustomerWriteService void MakeCustomerPreferred(CustomerId) void ChangeCustomerLocale(CustomerId, NewLocale) void CreateCustomer(Customer) void EditCustomerDetails(CustomerDetails) --------- CustomerReadService Customer GetCustomer(CustomerId) CustomerSet GetCustomersWithName(Name) CustomerSet GetPreferredCustomers() --------- and that's it. No Task/mediator architectures, no event sourcing, nothing.
- hn_throwaway_99 6y agoWhile I totally agree with this, I feel like at this point it's a little like the gif/jif argument. I don't really care that much if the original author pronounces it like the peanut butter, almost nobody else does. With CQRS, this is pretty trivially implemented in GraphQL as GraphQL explicitly separates queries from mutations (this is my favorite blog on the topic, https://www.apollographql.com/blog/designing-graphql-mutations-e09de826ed97/ https://www.apollographql.com/blog/designing-graphql-mutatio... ), but for some reason nobody talks about it as CQRS when they implement it like that. In fact, in the past 5ish years I've only heard about CQRS in an architecture that also used event sourcing.
- victor106 6y agothis makes sense. see also https://cqrs.files.wordpress.com/2010/11/cqrs_documents.pdf https://cqrs.files.wordpress.com/2010/11/cqrs_documents.pdf
- ben7799 6y agoI've implemented this for scalability a few times. In practical terms where I had to use it we ended up with the query portion being fulfilled by a materialized view that organized & aggregated the data in a way to make the queries feasible in milliseconds instead of minutes or tens of minutes. This pattern can be very important with respect to enormous data sources in the cloud but can even be useful in smaller data sets in "on prem" applications. Like anything though if you try to apply it everywhere that's just bad architecture.
- adamcharnock 6y agoI have been developing CQRS-like systems in smaller businesses for a while now. I've also been drawing on Domain Driven Design (DDD) principles. It took more a bit of experimentation to find how best to apply this to smaller scale software, but the results have been very pleasing. I started by sketching out a plan on how I would rigorously apply DDD & CQRS to a problem, but in the full-knowledge that it would likely be overkill. I then dropped out components which I felt were unnecessary for my particular situation. The result has been something fairly lightweight, without too much boilerplate, and very reliable. The clients concerned have certainly found it to be a rock solid part of the core business process. The need for this did not come from any performance requirement, but because I felt there had a to be sensible way to architect SME business software. Especially in a startup environment where business needs and processes were often changing. I felt that DDD allowed me to actually model businesses processes in software, rather than requiring business needs to adapt to the software I was developing. This may well be old news to enterprise-scale developers, but as a freelancer coming from the smaller-company end of the spectrum this was a big deal. As part of this process I developed a Python library [1] to facilitate writing evented/RPCed systems (event sourced or otherwise), and wrote up some very rough architecture tips [2] as part of the documentation. They are pretty high level, but may be an interesting (somewhat opinionated) starting point. [1]: https://lightbus.org https://lightbus.org [2]: https://lightbus.org/dev/explanation/architecture-tips/ https://lightbus.org/dev/explanation/architecture-tips/ Edit: Link update
- AnHonestComment 6y agoI wouldn’t say it’s “old news” in the enterprise world — and it’s fascinating to hear how you use it in a different context.
- zweihan 6y agoI am currently a contributor to a system that uses CQRS + ES. We are a fintech company and the positives we have adopting it outweighs the negatives. We use Lagom framework to implement CQRS+ES over Akka Actors and related technologies. There's enough guardrails/guides within the Lagom framework and documentations that it's hard to screw up too badly. And IMO I think the different pieces fit together very nicely! Overall the tradeoffs in system complexity vs scalability isn't too shabby.
- barumi 6y agoCQRS+ES is more about event sourcing and not as much about CQRS. The hard part about the scheme is the CAP theorem stuff, not how operations are segregated into commands and queries.
- waynesonfire 6y agoJust saw this came in on my Elixir mailing list and wanted to share with the folks interested in the topic. "Using Event Sourcing and CQRS with Incident - Part 1" https://pedroassumpcao.ghost.io/event-sourcing-and-cqrs-using-incident-part-1/ https://pedroassumpcao.ghost.io/event-sourcing-and-cqrs-usin...
- solumos 6y agoFiverr wrote a few blog posts about their implementation a while back: https://medium.com/fiverr-engineering/fiverrs-microservices-evolution-part-1-ee366bd0b420 https://medium.com/fiverr-engineering/fiverrs-microservices-... https://medium.com/fiverr-engineering/fiverrs-microservices-evolution-part-2-7bbcd6ceabda https://medium.com/fiverr-engineering/fiverrs-microservices-...
- bamazizi 6y agoI've been using CQRS pattern since 2017. Let me tell you it's not a complete system pattern, only a starting point. You need to design your system as LEGO pieces. What do you need to add? 1) Internal design guide/rule/law system so everyone understands and abides by. i.e. "C" command services should NOT talk to other command services, only to query services. But query services can only talk to other query services. You need service mesh and discovery. A bit complex. 2) Sink or aggregator. It's not efficient to have a query service listen to both events from message bus to store in DB and serve requests. It's best to have dedicated sink service(s) where data is streamed to database(s) with certain level of guarantees. 3) Rules engines or workflow system. Since command services cannot talk to one another, you'd need to orchestrate actions among them, that's what workflow systems are designed for. They connect to both query and command services and make sure long or complex tasks get done without the need to do hacky stuff! 4) Sanity! The moment you go with CQRS + Sink + Workflows, it'll become easy to feel overwhelmed and loose control. Start with a small set of MACRO based micro-services. Jam pack all commands into one service, same for query and sink and slowly dissect into multiple smaller services over time as workload grows and you need scalability (chaos monkey). This way you have only ~4 microservices to manage, well 5 if you need an API gateway. 5) With all the services and probably technologies, you need a contractual way to communicate. You need a system where CODE is the DOC! You need Protobuf or similar to design your schemas and api. You need GRPC because of protobuf...
- stingraycharles 6y ago> It's not efficient to have a query service listen to both events from message bus to store in DB and serve requests. It's best to have dedicated sink service(s) where data is streamed to database(s) with certain level of guarantees. Could you elaborate on this? We’ve deployed a view aggregate services that we expose through a HTTP interface, but you’re suggesting to stream this state to a database instead? Is that so that you can avoid the whole snapshotting trouble?
- bamazizi 6y agoMessage buses, as a source of truth, pass events around without much knowledge of its content relative to what applications need to process. They maintain history and offset of events based on topics. Sink/aggregators listen to events and, as per instructions, store payloads into db for efficient storage or optimized querying. The query service is only connecting to that db to execute queries. It's possible to have multiple dbs that store the same exact data but for different purposes. i.e. sql for normalized data or acid transactions and nosql for fast map reduce ... A properly designed system, treats dbs as disposable. Meaning, delete any db at anytime and just play back events from the beginning or from the last snapshot and rebuild a new db. Unfortunately making message buses the single source of truth is not that easy, and scary. Kafka is an expensive family relative, you better make sure you have the $$$ & the initial configuration is near perfect but still streaming requires some experts to help you get it right. Nats Steaming is good alternative to Kafka, I use and recommend it personally, but not perfect at high scale, like billions of events/day. Jetstream, successor to Nats Streaming, is coming but it's barely in preview stage and would need another year or so to be qualified for prod usage.
- rawgabbit 6y agoYou can use the database layer to achieve read-write separation as well. Martin Fowler mentioned reporting databases. With Azure Sql Database which is a SAAS you can pay for readable replicas. I have never used Google Spanner. But my understanding of Google Spanner database is it was designed from scratch so that reads never will slow down writes so it scales out of the box.
- twirlock 6y agoItt a bunch of people who have no clue about anything.