6 ms·

A word of caution for anyone considering an event-sourced architecture. I was on a large government project where the decision was made to use event sourcing, a

by lukev 7y ago

A word of caution for anyone considering an event-sourced architecture. I was on a large government project where the decision was made to use event sourcing, and it was disastrous. It ended up being a big contributor to several years of time and cost overruns. The reason is that for event sourcing to work, you have to have a pretty good idea of your application's requirements up front. It's simply not conducive to agile development, compared with using a traditional DB. The requirements were constantly shifting, and we were constantly realizing that we had the wrong semantics or structure for various fields, or that assumptions we had made about the coupling of different types of data simply didn't hold. This led to a ton of rewriting and churn on the view code, and required constant decisions on how to handle existing data in the "old" format. Some of the requirement churn even had ramifications for fundamental architectural characteristics such as support for atomic transactions, so there were several points at which we had to hack locking or other techniques to ensure consistency on top of the event sourcing approach. I do NOT recommend this, it turns everything into a huge mess. The worst part is, ultimately, the data sizes ended up not being that big. We could have run the whole thing off of append-only tables in a single beefy Postgres instance. Conclusion: if you are designing systems, you should definitely know what event sourcing is and the benefits it can provide. However, avoid it by default in favor of simpler more traditional models unless they are really infeasible for what you are trying to do. And then, lock down as many key requirements (at the very least, those around consistency and interop with other systems) as possible before charging ahead with implementation.

Yuioup 7y ago
Thank you for this. I've been warning people off Event Sourcing for a while now. The architecture is the most convoluted, pretentious, redundant and downright soul-crushing. If you see Event Sourcing anywhere, run away. Run far, far away.
- AndrewKemendo 7y ago
  I think you're doing it wrong. It shouldn't be any more complicated than writing events as a producer to a Kafka topic. Gets more complicated when you add in distributed computing, redundancy etc... However those things are always complicated
  - Yuioup 7y ago
    I got handed down an Event Sourcing project because the client left the original developer because he kept getting burn-outs. I can't shake off the project because it keeps getting stuck to the bottom of my shoe.
  - arkh 7y ago
    Simple problem: checking users have the rights to do something before you let them. Do you reconstruct the user account every time you want to check their rights (so, for every action they do)?
    - Autowired 7y ago
      No, you should not compute the state you need from the event log on every request, this would be absurd. Your authorization service can maintain its own database (a "view" of the current state), or even an in-memory representation computed at startup, and update it whenever a new event pops up. Alternatively, if you are using Kafka, you can use stuff like KTables to do this.
      - GordonS 7y ago
        This sounds like working around event sourcing - what value does event sourcing add here, vs simply not using it at all for user accounts?
        m45t3r 7y ago
        For this specific case, nothing, however let's say you need to know when the user got a specific scope and from who. You probably can answer this question in SQL if you prepared your database to do it (i.e: a changeset table), however in an event sourcing architecture you would gain this information for free. However, just because you're using event sourcing doesn't mean you don't have a database with the current state of your entities.
        dgregd 7y ago
- inopinatus 7y ago
  Some of worlds most useful and powerful data structures are the projection of an event log. The tables of a RDBMS, the balanced writes of a SSD, and even the classic: double-entry book-keeping. Even the data stream of a TCP connection is a projection of events, which is why (and how) we can reconstruct them by replaying captured segments. So just because there are some lousy executions of a general architecture, doesn’t mean we should recoil from the basic idea. My takeaway is that successful event-sourced structures are crafted for the domain they represent. I’ve developed a couple for my own work, for very specific aspects of an application, and they work well in context. If your experience has been that a general-purpose ES framework leads to shitty, hard-to-maintain apps, I’d say that’s evidence for the corollary.
  - nvarsj 7y ago
    Just because a tool is powerful doesn’t mean it’s appropriate. There’s a reason most people should just use a DB and not a raw event log. The issue I have with ES proponents is they seem to all pretend there is no additional complexity that comes with it. I think ES is useful but not always and requires weighing the costs and benefits, and we need to be honest about it.
  - legulere 7y ago
    You could say the same about e.g. lock free algorithms. If you work on them you are usually just playing with some cool technology instead of solving the business requirements.
- ekvilibrist 7y ago
  I've only suggested event sourcing once, and that was for keeping physical warehouse inventory synced with orders. I.e. instead of having like a database table with a "available product count", we made a view that calculated "product stock minus pending orders" on the fly at any given timestamp. Very efficient, simple and easy to debug. But as a general system architectural pattern it does seem to create more issues than it solves. Just look at the MailService example in the article. Dude, just use a queue...
  - Mouse47 7y ago
    Good approach. Much of software engineering is applying principals like event sourcing judiciously. Why does the whole system need to be in one architecture?
andreareina 7y ago
I'd love to read about the original assumptions -> original design -> revised assumptions -> design invalidations if you care (and are able) to write.
- lukev 7y ago
  Unfortunately it was on a consulting project with an NDA.
AndrewKemendo 7y ago
I mean that conclusion is true of basically every complex system. That's the whole point of hacking together an MVP first, to see what really matters. However it's silly IMO to use that criteria as an example of why not to do event driven architecture. It's well understood in the eda community that you generally don't start with streaming systems unless you know you need it from the start.
- lukev 7y ago
  In this case it was shiny object syndrome on the part of a few early architects. The whole project could have been a Rails app. The lesson is less about event sourcing, and more about picking architectures in light of business needs rather than trying to be fancy.
rolling_roland 7y ago
This, this and third time this. I have the exact same experience. You should know the business requirements very well before you choose event sourcing and hope they don't start changing drastically.
futzbar 7y ago
Couldn't agree more. It was disastrous for us with no real benefits.
richardARPANET 7y ago
+1. My company consulted on a project where the use of event sourcing was THE reason the entire project (and the client company) failed. This was the event sourcing library being used for that https://github.com/johnbywater/eventsourcing https://github.com/johnbywater/eventsourcing
- GordonS 7y ago
  Was the actual library used a significant part of the problem, or was it the pattern of event sourcing itself that wasn't a good fit?
  - richardARPANET 7y ago
    Both. The library was slow and corrupted data and they were using regular databases along with the event-sourced stuff. That would never work, the whole system would have needed to be event sourced for it to have any chance of working. It was essentially a CRUD app with the need for history of changes (for a few tables), SQL was the correct way to do it.
  - deleted 7y ago
    [deleted]
  - jbywater 7y ago
    Neither. If Richard's statements were facts they would violate his NDAs. In fact, these statements are falsehoods, and seem designed only to cause damage. Also, he's referring to 2016! The only "problem" is that in 2016 I was physically assaulted at work, Richard had to leave work soon after, and has been denigrating my work since then. For example, here's a post from 2016, where it appears he had a multi-account conversation with himself: https://news.ycombinator.com/item?id=13129798 https://news.ycombinator.com/item?id=13129798 Three years later, he's still unable to get over it and move on. This feels like a kind of twisted sexual investment, so I have chosen not to reply. The Python eventsourcing library is excellent, and event sourcing is a great idea. The library is open source, so if Richard had discovered a bug that caused "corrupt data", he could have raised an issue on GitHub. But there hasn't been such a bug, and no such issue has been raised. Everybody can look through the mere 56 closed issues in the entire history of this successful project to see how many "corrupted data" bugs there have been (zero). https://github.com/johnbywater/eventsourcing/issues https://github.com/johnbywater/eventsourcing/issues The library is being used successfully in production. Some users have data stores with millions of stored domain events, and the library still works very fast. I didn't hear about anybody having billions of events yet, but I wouldn't expect much difference in performance, so long as the infrastructure has sufficient volume. In case the wrong impression is created by listening to Richard's rubbish, SQL and event sourcing aren't somehow incompatible. Event sourcing isn't somehow the opposite of "regular databases". The library works very well with SQL databases, through both Django ORM and SQLAlchemy. It also works with NoSQL databases. If you see posts like this in future, please ignore them, it's just fake news. My understanding is that Richard has diagnosed mental problems. But perhaps there's a kind of nominative determinism from the shortened version of his name? I don't know. At any rate, I've been keeping a log of these occasions, in case I need to call the police again.
ltorg 7y ago
I'm actually quite happy to see that other people are also facing similar problems with event-sourced microservices. The project that I'm on (currently working for a neo-bank) has been trying to use event-sourcing from the get go, and oh god, is it a mess. The project has been going on for awhile and some of the devs thought it a good idea to focus on scalability and all the other metrics that don't matter. As your data evolves and your schema changes, you'll have a mix of messages of both the old and new schema in the same topic. You now have to change your consumer services to make sure you can handle the new schema as well as the old schema if they are important (think reconsuming in Kafka). Your code then gets really ugly having to handle OldXXXEvent NewXXXEvent and loads of if-else statements sprinkled in the code base. Either that, or migrate your data to a new topic which is a one-off exercise that takes time which you'll then have to do for different environments. I'm sure event sourcing has its place but I'm not entirely convinced the approach is necessarily better than just plain ol' database which would have saved us more time and get our product out quicker.
- randolorian 7y ago
  https://leanpub.com/esversioning/read https://leanpub.com/esversioning/read
seer 7y ago
A way to deal with it that I've seen to work with Kafka is to make your messages not last very long (say about a week) and make them explicitly idempotent. So an "old" message being run would not affect the app negatively. And then you make every producer of messages able to reproduce all the messages it knows about. So if you have a new service and need historical data, you ask all of your dependencies to resend the data, and existing services should not be affected. Schema evolution is natural - old messages with old schemas don't last very long, and you can slowly migrate services to new schemas as needed. If you structure your events that each one holds all state of an entity, you could get removal of data easily for free, as any new message would overwrite old state. Though the whole thing has its own problems though - especially when you go into large amounts of messages/data. Keeping copies of those around if you make a lot of changes can be quite expensive. So the whole event sourcing thing looks to me as it needs a decade or two to mature so that tools are built and best practices established. It looks to me as basically a way to keep data for a lot of separate services in a fluid eventually consistent way. Wonder if Clojure's Datomic is already there. Haven't used it myself but have read promising things about it.
- randolorian 7y ago
  Best practices around this have already been established. Most if not all event stores - which Kafka is not - have a concept called 'position.' You save the position atomically along with whatever you did with the message. Then if you crash, you simply ask for all messages starting from that position. If you have a new service (or a new projection), your position is 0 so you get everything.
  - seer 7y ago
    That is indeed the case, and this is what Kafka calls unlimited retention topics. This however hampers schema evolution significantly as you're not allowed to make backwards incompatible changes. Or rather if you allow those changes, every service would need to be able to handle every schema throughout time. If you make the retention to only several days this would mean you can evolve your schema with even breaking changes. Consumers would need to support only schemas that are "active right now". If the retention period is long enough that it is safe to assume all the consumers have acted on all the old messages you can delete those messages. You would need a mechanism to replay them though, if that data is ever needed again, but it would just be in the latest schema.
joostdevries 7y ago
Your word of caution is important. For some projects it's not initially clear what the right data model and bounded contexts are. And for some projects it's just overhead. But the converse is also true: for some projects/systems it turns out that it being events first is the _only_ way for it to work. I've encountered that the last few years with systems in logistics that show an integral view across parties. The way we approached that is that we started with a standard system and refactored to event processing approach when the decomposition into bounded contexts was clear. So in that sense it's similar to the right way of approaching microservices: start out with a 'monolith' that you split up.

by lukev 7y ago

Yuioup 7y ago

Thank you for this. I've been warning people off Event Sourcing for a while now. The architecture is the most convoluted, pretentious, redundant and downright soul-crushing. If you see Event Sourcing anywhere, run away. Run far, far away.

AndrewKemendo 7y ago
I think you're doing it wrong. It shouldn't be any more complicated than writing events as a producer to a Kafka topic. Gets more complicated when you add in distributed computing, redundancy etc... However those things are always complicated
- Yuioup 7y ago
  I got handed down an Event Sourcing project because the client left the original developer because he kept getting burn-outs. I can't shake off the project because it keeps getting stuck to the bottom of my shoe.
- arkh 7y ago
  Simple problem: checking users have the rights to do something before you let them. Do you reconstruct the user account every time you want to check their rights (so, for every action they do)?
  - Autowired 7y ago
    No, you should not compute the state you need from the event log on every request, this would be absurd. Your authorization service can maintain its own database (a "view" of the current state), or even an in-memory representation computed at startup, and update it whenever a new event pops up. Alternatively, if you are using Kafka, you can use stuff like KTables to do this.
    - GordonS 7y ago
      This sounds like working around event sourcing - what value does event sourcing add here, vs simply not using it at all for user accounts?
      - m45t3r 7y ago
        For this specific case, nothing, however let's say you need to know when the user got a specific scope and from who. You probably can answer this question in SQL if you prepared your database to do it (i.e: a changeset table), however in an event sourcing architecture you would gain this information for free. However, just because you're using event sourcing doesn't mean you don't have a database with the current state of your entities.
        dgregd 7y ago
inopinatus 7y ago
Some of worlds most useful and powerful data structures are the projection of an event log. The tables of a RDBMS, the balanced writes of a SSD, and even the classic: double-entry book-keeping. Even the data stream of a TCP connection is a projection of events, which is why (and how) we can reconstruct them by replaying captured segments. So just because there are some lousy executions of a general architecture, doesn’t mean we should recoil from the basic idea. My takeaway is that successful event-sourced structures are crafted for the domain they represent. I’ve developed a couple for my own work, for very specific aspects of an application, and they work well in context. If your experience has been that a general-purpose ES framework leads to shitty, hard-to-maintain apps, I’d say that’s evidence for the corollary.
- nvarsj 7y ago
  Just because a tool is powerful doesn’t mean it’s appropriate. There’s a reason most people should just use a DB and not a raw event log. The issue I have with ES proponents is they seem to all pretend there is no additional complexity that comes with it. I think ES is useful but not always and requires weighing the costs and benefits, and we need to be honest about it.
- legulere 7y ago
  You could say the same about e.g. lock free algorithms. If you work on them you are usually just playing with some cool technology instead of solving the business requirements.
ekvilibrist 7y ago
I've only suggested event sourcing once, and that was for keeping physical warehouse inventory synced with orders. I.e. instead of having like a database table with a "available product count", we made a view that calculated "product stock minus pending orders" on the fly at any given timestamp. Very efficient, simple and easy to debug. But as a general system architectural pattern it does seem to create more issues than it solves. Just look at the MailService example in the article. Dude, just use a queue...
- Mouse47 7y ago
  Good approach. Much of software engineering is applying principals like event sourcing judiciously. Why does the whole system need to be in one architecture?

andreareina 7y ago

I'd love to read about the original assumptions -> original design -> revised assumptions -> design invalidations if you care (and are able) to write.

lukev 7y ago
Unfortunately it was on a consulting project with an NDA.

AndrewKemendo 7y ago

I mean that conclusion is true of basically every complex system. That's the whole point of hacking together an MVP first, to see what really matters. However it's silly IMO to use that criteria as an example of why not to do event driven architecture. It's well understood in the eda community that you generally don't start with streaming systems unless you know you need it from the start.

lukev 7y ago
In this case it was shiny object syndrome on the part of a few early architects. The whole project could have been a Rails app. The lesson is less about event sourcing, and more about picking architectures in light of business needs rather than trying to be fancy.

rolling_roland 7y ago

This, this and third time this. I have the exact same experience. You should know the business requirements very well before you choose event sourcing and hope they don't start changing drastically.

futzbar 7y ago

Couldn't agree more. It was disastrous for us with no real benefits.

richardARPANET 7y ago

+1. My company consulted on a project where the use of event sourcing was THE reason the entire project (and the client company) failed. This was the event sourcing library being used for that https://github.com/johnbywater/eventsourcing https://github.com/johnbywater/eventsourcing

GordonS 7y ago
Was the actual library used a significant part of the problem, or was it the pattern of event sourcing itself that wasn't a good fit?
- richardARPANET 7y ago
  Both. The library was slow and corrupted data and they were using regular databases along with the event-sourced stuff. That would never work, the whole system would have needed to be event sourced for it to have any chance of working. It was essentially a CRUD app with the need for history of changes (for a few tables), SQL was the correct way to do it.
- deleted 7y ago
  [deleted]
- jbywater 7y ago
  Neither. If Richard's statements were facts they would violate his NDAs. In fact, these statements are falsehoods, and seem designed only to cause damage. Also, he's referring to 2016! The only "problem" is that in 2016 I was physically assaulted at work, Richard had to leave work soon after, and has been denigrating my work since then. For example, here's a post from 2016, where it appears he had a multi-account conversation with himself: https://news.ycombinator.com/item?id=13129798 https://news.ycombinator.com/item?id=13129798 Three years later, he's still unable to get over it and move on. This feels like a kind of twisted sexual investment, so I have chosen not to reply. The Python eventsourcing library is excellent, and event sourcing is a great idea. The library is open source, so if Richard had discovered a bug that caused "corrupt data", he could have raised an issue on GitHub. But there hasn't been such a bug, and no such issue has been raised. Everybody can look through the mere 56 closed issues in the entire history of this successful project to see how many "corrupted data" bugs there have been (zero). https://github.com/johnbywater/eventsourcing/issues https://github.com/johnbywater/eventsourcing/issues The library is being used successfully in production. Some users have data stores with millions of stored domain events, and the library still works very fast. I didn't hear about anybody having billions of events yet, but I wouldn't expect much difference in performance, so long as the infrastructure has sufficient volume. In case the wrong impression is created by listening to Richard's rubbish, SQL and event sourcing aren't somehow incompatible. Event sourcing isn't somehow the opposite of "regular databases". The library works very well with SQL databases, through both Django ORM and SQLAlchemy. It also works with NoSQL databases. If you see posts like this in future, please ignore them, it's just fake news. My understanding is that Richard has diagnosed mental problems. But perhaps there's a kind of nominative determinism from the shortened version of his name? I don't know. At any rate, I've been keeping a log of these occasions, in case I need to call the police again.

ltorg 7y ago

I'm actually quite happy to see that other people are also facing similar problems with event-sourced microservices. The project that I'm on (currently working for a neo-bank) has been trying to use event-sourcing from the get go, and oh god, is it a mess. The project has been going on for awhile and some of the devs thought it a good idea to focus on scalability and all the other metrics that don't matter. As your data evolves and your schema changes, you'll have a mix of messages of both the old and new schema in the same topic. You now have to change your consumer services to make sure you can handle the new schema as well as the old schema if they are important (think reconsuming in Kafka). Your code then gets really ugly having to handle OldXXXEvent NewXXXEvent and loads of if-else statements sprinkled in the code base. Either that, or migrate your data to a new topic which is a one-off exercise that takes time which you'll then have to do for different environments. I'm sure event sourcing has its place but I'm not entirely convinced the approach is necessarily better than just plain ol' database which would have saved us more time and get our product out quicker.

randolorian 7y ago
https://leanpub.com/esversioning/read https://leanpub.com/esversioning/read

seer 7y ago

A way to deal with it that I've seen to work with Kafka is to make your messages not last very long (say about a week) and make them explicitly idempotent. So an "old" message being run would not affect the app negatively. And then you make every producer of messages able to reproduce all the messages it knows about. So if you have a new service and need historical data, you ask all of your dependencies to resend the data, and existing services should not be affected. Schema evolution is natural - old messages with old schemas don't last very long, and you can slowly migrate services to new schemas as needed. If you structure your events that each one holds all state of an entity, you could get removal of data easily for free, as any new message would overwrite old state. Though the whole thing has its own problems though - especially when you go into large amounts of messages/data. Keeping copies of those around if you make a lot of changes can be quite expensive. So the whole event sourcing thing looks to me as it needs a decade or two to mature so that tools are built and best practices established. It looks to me as basically a way to keep data for a lot of separate services in a fluid eventually consistent way. Wonder if Clojure's Datomic is already there. Haven't used it myself but have read promising things about it.

randolorian 7y ago
Best practices around this have already been established. Most if not all event stores - which Kafka is not - have a concept called 'position.' You save the position atomically along with whatever you did with the message. Then if you crash, you simply ask for all messages starting from that position. If you have a new service (or a new projection), your position is 0 so you get everything.
- seer 7y ago
  That is indeed the case, and this is what Kafka calls unlimited retention topics. This however hampers schema evolution significantly as you're not allowed to make backwards incompatible changes. Or rather if you allow those changes, every service would need to be able to handle every schema throughout time. If you make the retention to only several days this would mean you can evolve your schema with even breaking changes. Consumers would need to support only schemas that are "active right now". If the retention period is long enough that it is safe to assume all the consumers have acted on all the old messages you can delete those messages. You would need a mechanism to replay them though, if that data is ever needed again, but it would just be in the latest schema.

joostdevries 7y ago

Your word of caution is important. For some projects it's not initially clear what the right data model and bounded contexts are. And for some projects it's just overhead. But the converse is also true: for some projects/systems it turns out that it being events first is the _only_ way for it to work. I've encountered that the last few years with systems in logistics that show an integral view across parties. The way we approached that is that we started with a standard system and refactored to event processing approach when the decomposition into bounded contexts was clear. So in that sense it's similar to the right way of approaching microservices: start out with a 'monolith' that you split up.