4 ms·

Didn't read the article but I'm in the process of eradicating Event Sourcing from a codebase and returning to the classical ACID database model. The boneheaded

Didn't read the article but I'm in the process of eradicating Event Sourcing from a codebase and returning to the classical ACID database model. The boneheaded decisions made by our predecessors is staggering and choosing to use Event Sourcing for everything is the dumbest of them all.

politician 3y ago
What are the major deficiencies of ES in that particular codebase? If you could drop some specifics in bullets that would be really helpful for me. I promise not to ambush you with apologetics.
- CodeCompost 3y ago
  This system was developed from 2008-2013, a very different time when hardware was "cheap" and the Cloud was not not a thing. Event Sourcing dictates that Events are never deleted which means that the data volume keeps growing and growing. There is - in this system - no way to delete old events. When I brought this up, the response was "Just add another hard drive". In the modern Cloud era, adding a hard drive is extremely expensive. The system uses CQRS and all events generate reports that are stored in Elasticsearch. Data is never deleted, only an extra event gets added /saying/ it's deleted. The data is still cached in Elasticsearch. All of it, all the data back to 2013. Added an extra 32GB of RAM just to keep up with it is ludicrously expensive. We're in Europe. Guess what a system like this does to GDPR. Can you tell me which events I need to delete when somebody says they want to be forgotten? Yeah. I can't delete old data. It's impossible without collapsing the entire system like a house of cards. Finally, and this is the piece de resistance, the developers decided to develop a relational database structure ON TOP OF EVENT SOURCING. We're talking primary keys, foreign keys, cascading and non-cascading deletes. Importing an Excel sheet of 10000 rows takes TWO WEEKS because it generates hundreds of events per Excel cell that is being read. We brought it down to 10 minutes and there is plenty of room for improvement it's just that we have other priorities right now. Currently, simple flat "tables" with no foreign keys take several seconds to import (just like in a regular RDMS). Oh yeah note that this is a system that is used by 4-5 users at a time, not hundreds or thousands of users.
capableweb 3y ago
Ok? Not sure what you want to talk about here, you probably need to give us a bit more context, especially if you even acknowledge you haven't even opened up the article to talk about the submission itself... Sometimes, the situation when something gets created and designed, looks very different from the current situation you're in N years later. So what might have looked like a boneheaded decision, could have been the best decision at that point. But us engineers like to lament our predecessors' code, I'm guilty of this sometimes too. But I try to remember that I don't have the full context of how things were when the code was initially written.
- mrkeen 3y ago
  > But I try to remember that I don't have the full context of how things were when the code was initially written. Ironically, that's what event-sourcing is for. If the facts were kept, then a better decision can be made today.
  - capableweb 3y ago
    Ironically, You missed the point of my comment :) I'm talking about things outside of code and certainly not about storing data about domain-specific things.
- deleted 3y ago
  [deleted]
lolinder 3y ago
I worked on a project where we decided that event sourcing was the way to go for a variety of legitimate business needs. We then implemented it with ACID transactions—an application-level framework writes the event to the Postgres database and in the same transaction updates all the computed views. At the scale we were working, this was totally fine performance-wise. Most people who have had bad experiences with event sourcing were actually having bad experiences with eventual consistency. All that event sourcing means is that you treat the events as the source of truth and everything else as computed from those events (and you could theoretically recompute it all at any time). Eventual consistency is an implementation detail and not a necessary one: you can implement event sourcing in a single Excel file if need be.
hot_gril 3y ago
Just beware, if you're using Postgres or MySQL, it's not fully ACID (specifically "I") unless you run xacts in serializable mode.
- deleted 3y ago
  [deleted]
deleted 3y ago
[deleted]

politician 3y ago

What are the major deficiencies of ES in that particular codebase? If you could drop some specifics in bullets that would be really helpful for me. I promise not to ambush you with apologetics.

CodeCompost 3y ago
This system was developed from 2008-2013, a very different time when hardware was "cheap" and the Cloud was not not a thing. Event Sourcing dictates that Events are never deleted which means that the data volume keeps growing and growing. There is - in this system - no way to delete old events. When I brought this up, the response was "Just add another hard drive". In the modern Cloud era, adding a hard drive is extremely expensive. The system uses CQRS and all events generate reports that are stored in Elasticsearch. Data is never deleted, only an extra event gets added /saying/ it's deleted. The data is still cached in Elasticsearch. All of it, all the data back to 2013. Added an extra 32GB of RAM just to keep up with it is ludicrously expensive. We're in Europe. Guess what a system like this does to GDPR. Can you tell me which events I need to delete when somebody says they want to be forgotten? Yeah. I can't delete old data. It's impossible without collapsing the entire system like a house of cards. Finally, and this is the piece de resistance, the developers decided to develop a relational database structure ON TOP OF EVENT SOURCING. We're talking primary keys, foreign keys, cascading and non-cascading deletes. Importing an Excel sheet of 10000 rows takes TWO WEEKS because it generates hundreds of events per Excel cell that is being read. We brought it down to 10 minutes and there is plenty of room for improvement it's just that we have other priorities right now. Currently, simple flat "tables" with no foreign keys take several seconds to import (just like in a regular RDMS). Oh yeah note that this is a system that is used by 4-5 users at a time, not hundreds or thousands of users.

capableweb 3y ago

Ok? Not sure what you want to talk about here, you probably need to give us a bit more context, especially if you even acknowledge you haven't even opened up the article to talk about the submission itself... Sometimes, the situation when something gets created and designed, looks very different from the current situation you're in N years later. So what might have looked like a boneheaded decision, could have been the best decision at that point. But us engineers like to lament our predecessors' code, I'm guilty of this sometimes too. But I try to remember that I don't have the full context of how things were when the code was initially written.

mrkeen 3y ago
> But I try to remember that I don't have the full context of how things were when the code was initially written. Ironically, that's what event-sourcing is for. If the facts were kept, then a better decision can be made today.
- capableweb 3y ago
  Ironically, You missed the point of my comment :) I'm talking about things outside of code and certainly not about storing data about domain-specific things.
deleted 3y ago
[deleted]

lolinder 3y ago

I worked on a project where we decided that event sourcing was the way to go for a variety of legitimate business needs. We then implemented it with ACID transactions—an application-level framework writes the event to the Postgres database and in the same transaction updates all the computed views. At the scale we were working, this was totally fine performance-wise. Most people who have had bad experiences with event sourcing were actually having bad experiences with eventual consistency. All that event sourcing means is that you treat the events as the source of truth and everything else as computed from those events (and you could theoretically recompute it all at any time). Eventual consistency is an implementation detail and not a necessary one: you can implement event sourcing in a single Excel file if need be.

hot_gril 3y ago

Just beware, if you're using Postgres or MySQL, it's not fully ACID (specifically "I") unless you run xacts in serializable mode.

deleted 3y ago
[deleted]

deleted 3y ago

[deleted]