17 ms·

Building a CQRS/ES web application in Elixir using Phoenix

181 points by tortilla 10y ago

arcbyte 10y ago
> CQRS library You missed the whole point of CQRS
- bglusman 10y ago
  Would you care to expand on what "the whole point" of CQRS is, or how they missed it? You seem to imply that the existence or creation of a library to support the pattern is, in some sense, against the spirit of the pattern, which is a) not at all obvious to me, and b) seems like something you should expand upon if you're criticising the article and/or author for HN's civil dialogue guidelines.
  - osd 10y ago
    This is a guess, but if I were to have a problem with the term CQRS Library, it is that the query and command sides of CQRS really shouldn't be linked in any way. CQRS is a methodology that stresses that decoupling above all else. If your the command and query portions of your CQRS tech stack both fall under a single library, that seems to go against the methodology.
  - realntl1 10y ago
    I think CQRS is more commonly viewed as an architectural pattern, not a code pattern, despite many definitions floating out there on the internet that focus on command/query object patterns. See this post: http://udidahan.com/2009/12/09/clarified-cqrs/ http://udidahan.com/2009/12/09/clarified-cqrs/. If we view it as an architectural pattern, it becomes folly to think that CQRS can be distilled into a library.
    - sb8244 10y ago
      > it becomes folly to think that CQRS can be distilled into a library Sincerely curious about this statement. I understand that CQRS is an architectural pattern, but couldn't a library implement that architecture and then provide ways for you to implement into that architecture? You don't get to pick the architecture nuances at that point, though.
      - karmajunkie 10y ago
        At its most basic level, CQRS is frankly too simple to take on a dependency as a cornerstone of a project. You can write a single base class that handles all of the responsibilities needed. Its NOT an architecture—its an architectural tool.
        kellyleahy 10y ago
        Agreed. This is so true. I've never ended up with a "library" for doing this. At my previous employer we had three different CQRS-based systems and two of the three of them even had completely different implementations for 90% of the CQRS infrastructure. They are both successful implementations and there was no need to share that bit of the code, as the two teams doing things came to different conclusions on how they wanted to write their CQRS code. The only thing we shared was the event store and some of the minor things on top of the event store.
      - Terr_ 10y ago
        IMO it's a bit like talking about a "Facade Framework". Using facades to encapsulate multiple complex objects behind a simpler interface may be an important of your overall design, but even if you use them it a lot you probably don't need (or want) to build a "Facade Framework" that lets you instantly define new ones with a few lines of meta-code. Any code you make for reuse by other people is going to contain something more in order to have value to them. For example, "A Facade library for dealing with various cloud services". Relating it back to CRQS, compare "CQRS Framework" to "A CQRS framework for command-line applications" or "A CQRS framework for websites". The CQRS-ness is a quality that can't stand just on its own.
- lucisferre 10y ago
  I know Greg personally, and I have heard him say things like "you don't need a library" in the past. However despite the fact that the basics are simple there are a lot of repeatable concepts that make sense to centralize around and even standardize. There is nothing wrong with a library, just as it is possible to do it without. One thing that absolutely needs library/drivers is persistence of the event store. I say this having built my own as well as contributing to open source ones in the past. It is still a hard and not well solved problem to do this well. Again it can be simple, but in the real world it isn't usually that simple. Tools are useful, tools are important. If CQRS/ES wants to make it into mainstream it needs to think about tooling. So far I have not been impressed with the people in the community hand waving about this and not having much to show for it.
  - slashdotdash 10y ago
    > If CQRS/ES wants to make it into mainstream it needs to think about tooling. I agree with this entirely. CQRS/ES applications really help from tailored tooling. Tracking chains of events using correlation and causation identifiers. Auditing commands. Projecting events out to datastores like Elasticsearch. Then using Kibana[1] to create dashboards to drill down into the data. I'd like to go into this in more detail in the future. Revisit this case study after 6-12 months and describe the pitfalls, and countermeasures. Identify tools that allow you to monitor the system proactively. [1] https://www.elastic.co/products/kibana https://www.elastic.co/products/kibana
  - jgeraert 10y ago
    > One thing that absolutely needs library/drivers is persistence of the event store. I say this having built my own as well as contributing to open source ones in the past. It is still a hard and not well solved problem to do this well. Again it can be simple, but in the real world it isn't usually that simple. Would it also be such a hard thing to do if you can delegate the actual persistence to something like a rdbms? What are the typical pitfalls?
    - dm3 10y ago
      Having built a few RDBMS-based event stores - it's pretty easy to do. There are two parts that I can think of that are not completely trivial - appending events to the stream while respecting "expected version" modifiers with optimal concurrency and allowing for fast and light subscriptions to new events (e.g. LISTEN/NOTIFY in Postgres, ringbuffer on the client). The good thing is that you can adapt the event store to your performance requirements and do the simplest thing possible in a huge amount of cases.
- slashdotdash 10y ago
  I'm the original author. Any application built following the CQRS/ES pattern requires development of the same building blocks: command registration and dispatch; hosting and delegation to aggregate roots; event handling; long running process managers. I built Commanded as a self-contained, reusable, open-source library. With the goal of demonstrating one approach to implementing the pattern using Elixir. I hope it provides some use. That's why I've written up the case study. Anyone can take the code as is, to bootstrap their own application. Use it as a learning tool, take away the good ideas. Rebuild and improve upon the bad. I received a pull request only today adding support for Greg Young's event store. That's going to broaden the appeal.
  - itsthecourier 10y ago
    Sounds great. By the way, in a common CRUD app with an entity having 30 fields, you basically: used a DTO, modified its data and merged in the DB row. Are we supposed to create 30 CQRS commands for this? Like property1updated, property2updated, etc.?
eric_b 10y ago
I have worked on, or cleaned up, 4 different CQRS/ES projects. They have all failed. Each time the people leading the project and championing the architecture were smart, capable, technically adept folks, but they couldn't make it work. There's more than one flavor of this particular arch, but Event Sourcing in general is simply not very useful for most projects. I'm sure there are use cases where it shines, but I have a hard time thinking of any. Versioning events, projection, reporting, maintenance, administration, dealing with failures, debugging, etc etc are all more challenging than with a traditional approach. Two of the projects I worked on used Event Store. That was one of the least production ready data stores I've encountered (the other being Datomic). I see a lot of excitement about CQRS/ES every two years or so (since 2010) and I strongly believe it is the wrong choice for just about every application.
- devbug 10y ago
  Do you mind detailing your experience with Dataomic? We're looking at ways to store 6-ary tuples (quadstore + temporality of existence and temporality of observation) of facts and build indices on top of them. I'm hesitant to move to (relatively) obscure data store without a really good idea of where that puts us.
  - eric_b 10y ago
    We didn't get terribly far down the road with Datomic. Management/administration of the DB was not for the faint of heart. (At the time the docs were simply terrible, maybe they are better now?) We would see data corruption/loss as well. We weren't doing anything terribly complicated with it, and data loss doing the simplest things was unacceptable. (It's entirely likely we were the cause of the data loss somehow, but we followed the guides and did nothing crazy or weird). So anyways, we just threw it out almost immediately. Conceptually it wasn't really a fit with the team or our application either. The query model and direct reading from the storage medium seemed silly - why not just use underlying storage medium directly for everything? (Besides, now you have to administer both Datomic and Cassandra/Oracle whatever) Not worth it for the obtuse query approach and limited value of "never forgetting" imo :)
bebop 10y ago
If the writer of the blog post is here, the link to the Elm language looks like it is missing an http:// http://
deleted 10y ago
[deleted]
sbellware 10y ago
I've done a number of event-sourced projects, and have built a good amount of tooling over time to support the work. There's one overarching issue I see early on in the adoption process. The difference between what is being called "traditional" architecture here and event sourcing is in which things that have to be correct from the very start. I.e.: The things that you can change later in traditional vs event-sourced are different. If you don't know realize that this is the case, and proceed with developing an event-sourced system, you'll end up facing decisions that you'd expected to be able to reverse later which are not reversible. Architecture at its root is concerned primarily with "reversibility" - understanding which decisions can be reversed and which can't and making sure that you get the irreversible decisions correct up-front, and focus design efforts on them. If you've only ever built systems under a single paradigm, CRUD for example, then you might not even be aware of the differences in reversibility concerns between paradigms. I know folks who've succeeded wonderfully with event sourcing and I know folks who've failed. I don't know anyone who has failed with event sourcing who were not themselves to blame for the failure. Some of those have refused to consider the possibility that they are the failure's root cause, and then derided others who point that out, as has been done here in this discussion. Event sourcing is a big leap. There is a lot of presumed knowledge that isn't easily spelled out in bite-sized chunks like blog posts and tweets. You won't learn it (and wield it safely) if you believe that all things should be as readily-consumed as Meteor over Mongo, or Rails, or pick-your-forms-over-data tool. If you don't work to become aware of the dominant paradigm that shapes your preconceptions, then it will be next to impossible to leverage another paradigm without polluting it with concerns that are counterproductive. The things that need the application of your intellect in event sourcing are not the things that need the application of your intellect in "traditional" (or any other) architectural paradigm. Square pegs, round holes, etc. But since our perception of the squareness of the pegs is filtered through the lens of our own predispositions, we may not even know that we're in the process of attempting to force-fit things that need a tight fit, and that will cause project failure without the tight fit. Event sourcing makes things simple - far simpler than ORM, for example. But it only does so when approaching it from its own predispositions, rather than the predispositions (especially unrecognized) of some foreign and impertinent approach. I find the prospect of working on "traditional" systems depressing now. I find having to solve problems that shouldn't exist in the first place to be soul-crushing. I have an expectation of productivity that is sustained and sustainable over the long haul, and doesn't decrease even as the size of the system/team/complexity and expectations grow. But, I've been following event sourcing since it's unnamed, prototypical forms in 2006. I started putting it into practice many years later. I didn't expect it to come to me fully formed after watching a handful of conference presentations, and I didn't expect it to come to me as easily as other architectural paradigms and approaches I've used in the past. I'm still experiencing new realizations about event sourcing. The journey isn't done yet, but I'm far better off than I was. I've compared notes with colleagues who are also quite deep into the transition and have heard similar observations repeated back to me: I can do an event-sourced system today as fast as I could have built the same system in a rapid-prototyping tool like Rails in years past. However, unlike Rails (for example), my productivity, and my team's productivity remains stable, and changes in scale and complexity of the business and its organization don't induce the panic that it used to. And my expectation for the quality and the caliber of the implementation has only increased - and dramatically so relative to the implementations I still occasionally see in "traditional" systems. But had I not had to put some distance between my own mind and its hard-won-yet-entrenched preconceptions I might not have seen the instincts and subconscious mechanisms which surely would have led me to underestimating how and where I needed to focus in order to not build elaborate structures that I would have been crushed by in the end. If anyone is really and truly interested in digging into event sourcing, and really seeing it through its own lens (by becoming aware of the existing lens we all may have), I'd love to help out. Yes, I make money doing this, and yes, I have a vested interest - but I also spend a lot of volunteer time with devs who are earnestly trying to get to the point where event-sourcing and all of its unfamiliar challenges are just another thing that was learned and facility cultivated over time. I can also help ween you off a compulsion to adopt event sourcing for a project that you want to do as fast as you can with a paradigm that you do already have a grasp of. Either way. Ultimately, I find the potential of a future with event sourcing far brighter than one without, but I've had to cross many bridges and dismantle many self-imposed obstructions to get there. I hope that others are open to these kinds of experiences and that we have more profound conversations about event sourcing experience reports from the other side of the chasm. My 2 cents, anyway.
privacyfornow 10y ago
One way to understand CQRS is to realize that even the traditional model relies on CQRS underneath, it just so happens that the relational databases hide some of this behind a blocking wait on commit while a background thread reads and flushes the event log (consisting of SQL statements written to a redo log for example). Really what you are gaining by inverting the database like this is a lot more flexibility and simplicity for initial development and subsequent change management (maintenance). The flexibility I speak of manifests in your ability to tweak availability by introducing controllable circuit breakers with availability at the expense of some latency or alternatively putting an upper bound on latency at the expense of some availability (with every tradeoff fully documented and demonstrably adhered to).
maxekman 10y ago
Very interesting read, and more so the comments here with all the various experiences. I'm the author of Event Horizon for Go [1] which is currently being developed along side a project that uses it, in this case a CRM-like system where the audit log is a selling point. We still have a lot of unsolved areas but as some have noted it is a very rewarding developer experience. How it behaves in production is still to be seen, but reading here has given me a few good insights to prepare. [1] https://github.com/looplab/eventhorizon https://github.com/looplab/eventhorizon

181 points by tortilla 10y ago

arcbyte 10y ago

> CQRS library You missed the whole point of CQRS

bglusman 10y ago
Would you care to expand on what "the whole point" of CQRS is, or how they missed it? You seem to imply that the existence or creation of a library to support the pattern is, in some sense, against the spirit of the pattern, which is a) not at all obvious to me, and b) seems like something you should expand upon if you're criticising the article and/or author for HN's civil dialogue guidelines.
- osd 10y ago
  This is a guess, but if I were to have a problem with the term CQRS Library, it is that the query and command sides of CQRS really shouldn't be linked in any way. CQRS is a methodology that stresses that decoupling above all else. If your the command and query portions of your CQRS tech stack both fall under a single library, that seems to go against the methodology.
- realntl1 10y ago
  I think CQRS is more commonly viewed as an architectural pattern, not a code pattern, despite many definitions floating out there on the internet that focus on command/query object patterns. See this post: http://udidahan.com/2009/12/09/clarified-cqrs/ http://udidahan.com/2009/12/09/clarified-cqrs/. If we view it as an architectural pattern, it becomes folly to think that CQRS can be distilled into a library.
  - sb8244 10y ago
    > it becomes folly to think that CQRS can be distilled into a library Sincerely curious about this statement. I understand that CQRS is an architectural pattern, but couldn't a library implement that architecture and then provide ways for you to implement into that architecture? You don't get to pick the architecture nuances at that point, though.
    - karmajunkie 10y ago
      At its most basic level, CQRS is frankly too simple to take on a dependency as a cornerstone of a project. You can write a single base class that handles all of the responsibilities needed. Its NOT an architecture—its an architectural tool.
      - kellyleahy 10y ago
        Agreed. This is so true. I've never ended up with a "library" for doing this. At my previous employer we had three different CQRS-based systems and two of the three of them even had completely different implementations for 90% of the CQRS infrastructure. They are both successful implementations and there was no need to share that bit of the code, as the two teams doing things came to different conclusions on how they wanted to write their CQRS code. The only thing we shared was the event store and some of the minor things on top of the event store.
    - Terr_ 10y ago
      IMO it's a bit like talking about a "Facade Framework". Using facades to encapsulate multiple complex objects behind a simpler interface may be an important of your overall design, but even if you use them it a lot you probably don't need (or want) to build a "Facade Framework" that lets you instantly define new ones with a few lines of meta-code. Any code you make for reuse by other people is going to contain something more in order to have value to them. For example, "A Facade library for dealing with various cloud services". Relating it back to CRQS, compare "CQRS Framework" to "A CQRS framework for command-line applications" or "A CQRS framework for websites". The CQRS-ness is a quality that can't stand just on its own.
lucisferre 10y ago
I know Greg personally, and I have heard him say things like "you don't need a library" in the past. However despite the fact that the basics are simple there are a lot of repeatable concepts that make sense to centralize around and even standardize. There is nothing wrong with a library, just as it is possible to do it without. One thing that absolutely needs library/drivers is persistence of the event store. I say this having built my own as well as contributing to open source ones in the past. It is still a hard and not well solved problem to do this well. Again it can be simple, but in the real world it isn't usually that simple. Tools are useful, tools are important. If CQRS/ES wants to make it into mainstream it needs to think about tooling. So far I have not been impressed with the people in the community hand waving about this and not having much to show for it.
- slashdotdash 10y ago
  > If CQRS/ES wants to make it into mainstream it needs to think about tooling. I agree with this entirely. CQRS/ES applications really help from tailored tooling. Tracking chains of events using correlation and causation identifiers. Auditing commands. Projecting events out to datastores like Elasticsearch. Then using Kibana[1] to create dashboards to drill down into the data. I'd like to go into this in more detail in the future. Revisit this case study after 6-12 months and describe the pitfalls, and countermeasures. Identify tools that allow you to monitor the system proactively. [1] https://www.elastic.co/products/kibana https://www.elastic.co/products/kibana
- jgeraert 10y ago
  > One thing that absolutely needs library/drivers is persistence of the event store. I say this having built my own as well as contributing to open source ones in the past. It is still a hard and not well solved problem to do this well. Again it can be simple, but in the real world it isn't usually that simple. Would it also be such a hard thing to do if you can delegate the actual persistence to something like a rdbms? What are the typical pitfalls?
  - dm3 10y ago
    Having built a few RDBMS-based event stores - it's pretty easy to do. There are two parts that I can think of that are not completely trivial - appending events to the stream while respecting "expected version" modifiers with optimal concurrency and allowing for fast and light subscriptions to new events (e.g. LISTEN/NOTIFY in Postgres, ringbuffer on the client). The good thing is that you can adapt the event store to your performance requirements and do the simplest thing possible in a huge amount of cases.
slashdotdash 10y ago
I'm the original author. Any application built following the CQRS/ES pattern requires development of the same building blocks: command registration and dispatch; hosting and delegation to aggregate roots; event handling; long running process managers. I built Commanded as a self-contained, reusable, open-source library. With the goal of demonstrating one approach to implementing the pattern using Elixir. I hope it provides some use. That's why I've written up the case study. Anyone can take the code as is, to bootstrap their own application. Use it as a learning tool, take away the good ideas. Rebuild and improve upon the bad. I received a pull request only today adding support for Greg Young's event store. That's going to broaden the appeal.
- itsthecourier 10y ago
  Sounds great. By the way, in a common CRUD app with an entity having 30 fields, you basically: used a DTO, modified its data and merged in the DB row. Are we supposed to create 30 CQRS commands for this? Like property1updated, property2updated, etc.?

eric_b 10y ago

I have worked on, or cleaned up, 4 different CQRS/ES projects. They have all failed. Each time the people leading the project and championing the architecture were smart, capable, technically adept folks, but they couldn't make it work. There's more than one flavor of this particular arch, but Event Sourcing in general is simply not very useful for most projects. I'm sure there are use cases where it shines, but I have a hard time thinking of any. Versioning events, projection, reporting, maintenance, administration, dealing with failures, debugging, etc etc are all more challenging than with a traditional approach. Two of the projects I worked on used Event Store. That was one of the least production ready data stores I've encountered (the other being Datomic). I see a lot of excitement about CQRS/ES every two years or so (since 2010) and I strongly believe it is the wrong choice for just about every application.

devbug 10y ago
Do you mind detailing your experience with Dataomic? We're looking at ways to store 6-ary tuples (quadstore + temporality of existence and temporality of observation) of facts and build indices on top of them. I'm hesitant to move to (relatively) obscure data store without a really good idea of where that puts us.
- eric_b 10y ago
  We didn't get terribly far down the road with Datomic. Management/administration of the DB was not for the faint of heart. (At the time the docs were simply terrible, maybe they are better now?) We would see data corruption/loss as well. We weren't doing anything terribly complicated with it, and data loss doing the simplest things was unacceptable. (It's entirely likely we were the cause of the data loss somehow, but we followed the guides and did nothing crazy or weird). So anyways, we just threw it out almost immediately. Conceptually it wasn't really a fit with the team or our application either. The query model and direct reading from the storage medium seemed silly - why not just use underlying storage medium directly for everything? (Besides, now you have to administer both Datomic and Cassandra/Oracle whatever) Not worth it for the obtuse query approach and limited value of "never forgetting" imo :)

bebop 10y ago

If the writer of the blog post is here, the link to the Elm language looks like it is missing an http:// http://

deleted 10y ago

[deleted]

sbellware 10y ago

I've done a number of event-sourced projects, and have built a good amount of tooling over time to support the work. There's one overarching issue I see early on in the adoption process. The difference between what is being called "traditional" architecture here and event sourcing is in which things that have to be correct from the very start. I.e.: The things that you can change later in traditional vs event-sourced are different. If you don't know realize that this is the case, and proceed with developing an event-sourced system, you'll end up facing decisions that you'd expected to be able to reverse later which are not reversible. Architecture at its root is concerned primarily with "reversibility" - understanding which decisions can be reversed and which can't and making sure that you get the irreversible decisions correct up-front, and focus design efforts on them. If you've only ever built systems under a single paradigm, CRUD for example, then you might not even be aware of the differences in reversibility concerns between paradigms. I know folks who've succeeded wonderfully with event sourcing and I know folks who've failed. I don't know anyone who has failed with event sourcing who were not themselves to blame for the failure. Some of those have refused to consider the possibility that they are the failure's root cause, and then derided others who point that out, as has been done here in this discussion. Event sourcing is a big leap. There is a lot of presumed knowledge that isn't easily spelled out in bite-sized chunks like blog posts and tweets. You won't learn it (and wield it safely) if you believe that all things should be as readily-consumed as Meteor over Mongo, or Rails, or pick-your-forms-over-data tool. If you don't work to become aware of the dominant paradigm that shapes your preconceptions, then it will be next to impossible to leverage another paradigm without polluting it with concerns that are counterproductive. The things that need the application of your intellect in event sourcing are not the things that need the application of your intellect in "traditional" (or any other) architectural paradigm. Square pegs, round holes, etc. But since our perception of the squareness of the pegs is filtered through the lens of our own predispositions, we may not even know that we're in the process of attempting to force-fit things that need a tight fit, and that will cause project failure without the tight fit. Event sourcing makes things simple - far simpler than ORM, for example. But it only does so when approaching it from its own predispositions, rather than the predispositions (especially unrecognized) of some foreign and impertinent approach. I find the prospect of working on "traditional" systems depressing now. I find having to solve problems that shouldn't exist in the first place to be soul-crushing. I have an expectation of productivity that is sustained and sustainable over the long haul, and doesn't decrease even as the size of the system/team/complexity and expectations grow. But, I've been following event sourcing since it's unnamed, prototypical forms in 2006. I started putting it into practice many years later. I didn't expect it to come to me fully formed after watching a handful of conference presentations, and I didn't expect it to come to me as easily as other architectural paradigms and approaches I've used in the past. I'm still experiencing new realizations about event sourcing. The journey isn't done yet, but I'm far better off than I was. I've compared notes with colleagues who are also quite deep into the transition and have heard similar observations repeated back to me: I can do an event-sourced system today as fast as I could have built the same system in a rapid-prototyping tool like Rails in years past. However, unlike Rails (for example), my productivity, and my team's productivity remains stable, and changes in scale and complexity of the business and its organization don't induce the panic that it used to. And my expectation for the quality and the caliber of the implementation has only increased - and dramatically so relative to the implementations I still occasionally see in "traditional" systems. But had I not had to put some distance between my own mind and its hard-won-yet-entrenched preconceptions I might not have seen the instincts and subconscious mechanisms which surely would have led me to underestimating how and where I needed to focus in order to not build elaborate structures that I would have been crushed by in the end. If anyone is really and truly interested in digging into event sourcing, and really seeing it through its own lens (by becoming aware of the existing lens we all may have), I'd love to help out. Yes, I make money doing this, and yes, I have a vested interest - but I also spend a lot of volunteer time with devs who are earnestly trying to get to the point where event-sourcing and all of its unfamiliar challenges are just another thing that was learned and facility cultivated over time. I can also help ween you off a compulsion to adopt event sourcing for a project that you want to do as fast as you can with a paradigm that you do already have a grasp of. Either way. Ultimately, I find the potential of a future with event sourcing far brighter than one without, but I've had to cross many bridges and dismantle many self-imposed obstructions to get there. I hope that others are open to these kinds of experiences and that we have more profound conversations about event sourcing experience reports from the other side of the chasm. My 2 cents, anyway.

privacyfornow 10y ago

One way to understand CQRS is to realize that even the traditional model relies on CQRS underneath, it just so happens that the relational databases hide some of this behind a blocking wait on commit while a background thread reads and flushes the event log (consisting of SQL statements written to a redo log for example). Really what you are gaining by inverting the database like this is a lot more flexibility and simplicity for initial development and subsequent change management (maintenance). The flexibility I speak of manifests in your ability to tweak availability by introducing controllable circuit breakers with availability at the expense of some latency or alternatively putting an upper bound on latency at the expense of some availability (with every tradeoff fully documented and demonstrably adhered to).

maxekman 10y ago

Very interesting read, and more so the comments here with all the various experiences. I'm the author of Event Horizon for Go [1] which is currently being developed along side a project that uses it, in this case a CRM-like system where the audit log is a selling point. We still have a lot of unsolved areas but as some have noted it is a very rewarding developer experience. How it behaves in production is still to be seen, but reading here has given me a few good insights to prepare. [1] https://github.com/looplab/eventhorizon https://github.com/looplab/eventhorizon