5 ms·

I've been using CQRS pattern since 2017. Let me tell you it's not a complete system pattern, only a starting point. You need to design your system as LEGO piece

I've been using CQRS pattern since 2017. Let me tell you it's not a complete system pattern, only a starting point. You need to design your system as LEGO pieces. What do you need to add? 1) Internal design guide/rule/law system so everyone understands and abides by. i.e. "C" command services should NOT talk to other command services, only to query services. But query services can only talk to other query services. You need service mesh and discovery. A bit complex. 2) Sink or aggregator. It's not efficient to have a query service listen to both events from message bus to store in DB and serve requests. It's best to have dedicated sink service(s) where data is streamed to database(s) with certain level of guarantees. 3) Rules engines or workflow system. Since command services cannot talk to one another, you'd need to orchestrate actions among them, that's what workflow systems are designed for. They connect to both query and command services and make sure long or complex tasks get done without the need to do hacky stuff! 4) Sanity! The moment you go with CQRS + Sink + Workflows, it'll become easy to feel overwhelmed and loose control. Start with a small set of MACRO based micro-services. Jam pack all commands into one service, same for query and sink and slowly dissect into multiple smaller services over time as workload grows and you need scalability (chaos monkey). This way you have only ~4 microservices to manage, well 5 if you need an API gateway. 5) With all the services and probably technologies, you need a contractual way to communicate. You need a system where CODE is the DOC! You need Protobuf or similar to design your schemas and api. You need GRPC because of protobuf...

stingraycharles 6y ago
> It's not efficient to have a query service listen to both events from message bus to store in DB and serve requests. It's best to have dedicated sink service(s) where data is streamed to database(s) with certain level of guarantees. Could you elaborate on this? We’ve deployed a view aggregate services that we expose through a HTTP interface, but you’re suggesting to stream this state to a database instead? Is that so that you can avoid the whole snapshotting trouble?
- bamazizi 6y ago
  Message buses, as a source of truth, pass events around without much knowledge of its content relative to what applications need to process. They maintain history and offset of events based on topics. Sink/aggregators listen to events and, as per instructions, store payloads into db for efficient storage or optimized querying. The query service is only connecting to that db to execute queries. It's possible to have multiple dbs that store the same exact data but for different purposes. i.e. sql for normalized data or acid transactions and nosql for fast map reduce ... A properly designed system, treats dbs as disposable. Meaning, delete any db at anytime and just play back events from the beginning or from the last snapshot and rebuild a new db. Unfortunately making message buses the single source of truth is not that easy, and scary. Kafka is an expensive family relative, you better make sure you have the $$$ & the initial configuration is near perfect but still streaming requires some experts to help you get it right. Nats Steaming is good alternative to Kafka, I use and recommend it personally, but not perfect at high scale, like billions of events/day. Jetstream, successor to Nats Streaming, is coming but it's barely in preview stage and would need another year or so to be qualified for prod usage.
  - stingraycharles 6y ago
    I see, so it’s a way of effectively decoupling the state of the aggregate (store it inside the db) with the actual querying (just query the db). That way, at any time, you can just stop the aggregate/sink, and recover where you left off. We have implemented our event store on top of PostgreSQL. I’m not a big fan of using Kafka as an event store.
Cieplak 6y ago
I would argue the most important “thing to add” here is a business model and company structure that actually benefits from this complexity. There are indeed businesses and large teams that benefit immensely from well-thought-out architectures such as the one you describe. Most internet businesses, though, just need a single service deployed across two or more availability zones, that talks to a SQL database with one or more hot standbys. Once that basic architecture stops working, you have good problems.
dsies 6y ago
Full disclosure: I co-founded a startup (https://batch.sh https://batch.sh) which focuses specifically on solving problems in the event driven space - we do event archive, search and replay. So this space is super interesting to me :) You are totally right - the CQRS pattern alone is not enough. In my experience, most shops that go down the CQRS path, also tend to make use of other event driven patterns like event sourcing. As with most things, there's pro's and con's and the most obvious con here is increased complexity in exchange for higher reliability and scale. Your advice is great, especially deciding on a structured message format such as protobuf. I would 100% avoid JSON schema as it's possible folks will forget to fill out fields. Protobuf has excellent support by now and there is a ton of support tooling for it. I'd be hard pressed to choose Avro at this point (unless you're a Java or Kafka shop that has it already in their ecosystem). In addition, if you are already on CQRS, I would advise looking into fully embracing event driven - by utilizing an event bus of some sort (rabbit, kafka, eventbridge), you could make message passing completely async (and avoid having to use gRPC, which is another layer of complexity, as you mentioned). A good friend of mine once said that in order to successfully do event driven arch, you must be OK with eventual consistency and I think that lies at the heart of this. If you are OK with eventual consistency, you understand the burden of complexity you're bringing on.
- mikhailfranco 6y ago
  Add MQTT to your lists of event buses. MQTT is lightweight and fits nicely into IoT sources, backend streaming and even scalable frontend distribution to browser or mobile clients. I recently did a quick'n'dirty PoC with paho.js and D3.js for live charting in the browser. In a pub-sub event-driven world, CQRS just means having distinct events for: Commands to do something; Query matches (CEP-style); and other Notifications that something has happened. Queries are compiled and unfolded into streaming operator graphs, delivering match events to the client. Queries and their operator graphs can be factored to share common sub-expressions (like the RETE rules algorithm). A good start-up idea would be MQTT 'appstreamer' framework, with model-driven message formats, validation and APIs, together with pluggable engines for CEP queries, rules, workflows, constraint solvers and other business logic.

stingraycharles 6y ago

> It's not efficient to have a query service listen to both events from message bus to store in DB and serve requests. It's best to have dedicated sink service(s) where data is streamed to database(s) with certain level of guarantees. Could you elaborate on this? We’ve deployed a view aggregate services that we expose through a HTTP interface, but you’re suggesting to stream this state to a database instead? Is that so that you can avoid the whole snapshotting trouble?

bamazizi 6y ago
Message buses, as a source of truth, pass events around without much knowledge of its content relative to what applications need to process. They maintain history and offset of events based on topics. Sink/aggregators listen to events and, as per instructions, store payloads into db for efficient storage or optimized querying. The query service is only connecting to that db to execute queries. It's possible to have multiple dbs that store the same exact data but for different purposes. i.e. sql for normalized data or acid transactions and nosql for fast map reduce ... A properly designed system, treats dbs as disposable. Meaning, delete any db at anytime and just play back events from the beginning or from the last snapshot and rebuild a new db. Unfortunately making message buses the single source of truth is not that easy, and scary. Kafka is an expensive family relative, you better make sure you have the $$$ & the initial configuration is near perfect but still streaming requires some experts to help you get it right. Nats Steaming is good alternative to Kafka, I use and recommend it personally, but not perfect at high scale, like billions of events/day. Jetstream, successor to Nats Streaming, is coming but it's barely in preview stage and would need another year or so to be qualified for prod usage.
- stingraycharles 6y ago
  I see, so it’s a way of effectively decoupling the state of the aggregate (store it inside the db) with the actual querying (just query the db). That way, at any time, you can just stop the aggregate/sink, and recover where you left off. We have implemented our event store on top of PostgreSQL. I’m not a big fan of using Kafka as an event store.

Cieplak 6y ago

I would argue the most important “thing to add” here is a business model and company structure that actually benefits from this complexity. There are indeed businesses and large teams that benefit immensely from well-thought-out architectures such as the one you describe. Most internet businesses, though, just need a single service deployed across two or more availability zones, that talks to a SQL database with one or more hot standbys. Once that basic architecture stops working, you have good problems.

dsies 6y ago

Full disclosure: I co-founded a startup (https://batch.sh https://batch.sh) which focuses specifically on solving problems in the event driven space - we do event archive, search and replay. So this space is super interesting to me :) You are totally right - the CQRS pattern alone is not enough. In my experience, most shops that go down the CQRS path, also tend to make use of other event driven patterns like event sourcing. As with most things, there's pro's and con's and the most obvious con here is increased complexity in exchange for higher reliability and scale. Your advice is great, especially deciding on a structured message format such as protobuf. I would 100% avoid JSON schema as it's possible folks will forget to fill out fields. Protobuf has excellent support by now and there is a ton of support tooling for it. I'd be hard pressed to choose Avro at this point (unless you're a Java or Kafka shop that has it already in their ecosystem). In addition, if you are already on CQRS, I would advise looking into fully embracing event driven - by utilizing an event bus of some sort (rabbit, kafka, eventbridge), you could make message passing completely async (and avoid having to use gRPC, which is another layer of complexity, as you mentioned). A good friend of mine once said that in order to successfully do event driven arch, you must be OK with eventual consistency and I think that lies at the heart of this. If you are OK with eventual consistency, you understand the burden of complexity you're bringing on.

mikhailfranco 6y ago
Add MQTT to your lists of event buses. MQTT is lightweight and fits nicely into IoT sources, backend streaming and even scalable frontend distribution to browser or mobile clients. I recently did a quick'n'dirty PoC with paho.js and D3.js for live charting in the browser. In a pub-sub event-driven world, CQRS just means having distinct events for: Commands to do something; Query matches (CEP-style); and other Notifications that something has happened. Queries are compiled and unfolded into streaming operator graphs, delivering match events to the client. Queries and their operator graphs can be factored to share common sub-expressions (like the RETE rules algorithm). A good start-up idea would be MQTT 'appstreamer' framework, with model-driven message formats, validation and APIs, together with pluggable engines for CEP queries, rules, workflows, constraint solvers and other business logic.