4 ms·

I have never seen a productive application of event sourcing at scale. The idea is super elegant but there’s so much extra overhead. Your data size is easily 10

I have never seen a productive application of event sourcing at scale. The idea is super elegant but there’s so much extra overhead. Your data size is easily 10x bigger, unless things nearly never change anyway (and then why do it, right?). Even simple things such as fixing a typo require a Command to be stored as an Event etc etc. Fixing fuckups by manual database edits / script runs gets harder because it means you’ll create a mismatch between the events and the main db, so you ought to inject events instead, but you’re under time pressure so what to do, and the list goes on. I wish these things weren’t issues because in a very real way event sourcing feels like the “obviously best way” to deal with pretty much any kind of structured data storage. I want it to succeed. Does anyone know of a system that successfully did/does event sourcing at scale? How did you deal with these downsides?

bob1029 3y ago
> Even simple things such as fixing a typo require a Command to be stored as an Event etc etc. Fixing fuckups ... Event sourcing is a completely different way of thinking about business information. Of course you are going to run into things that look like "downsides". Consider the desired semantics of event sourcing and how your concerns run directly in the face of these semantics. If you aren't comfortable with what qualitatively seem like "trivial" updates, then I don't think a large enough leap of faith has been made. We like to empathize with the machine but sometimes we take it a bit too far. Event sourcing absolutely works and it works at the most massive scales the best. If you can dedicate an entire wing of your 15th floor office to "enterprise messaging and events", this stuff 5000% works. It seems to be at the small scale that it fails most frequently. At small scale, it's usually just one developer who has enough motivation to see it through, and the moment they shift their focus it falls apart. I saw it work in manufacturing in ways I cannot do justice in HN comment size limits.
- turnsout 3y ago
  Yes, the event model is the only thing that makes sense. Just wanted to say, I don't think Kent's idea is in conflict with the Event Sourcing pattern, as long as these "Effective/Posting" events are stored append-only. Once you start updating the "Effective" date, you're losing data (a customer might call back and ask what the old Effective date was, to understand a paper bill, and you've lost it).
- bbkane 3y ago
  I would *love* to read a blog post/Google Doc/GitHub Gist/??? about this if you get the time to write one
cratermoon 3y ago
> Does anyone know of a system that successfully did/does event sourcing at scale? The git version control system. > Even simple things such as fixing a typo require a Command to be stored as an Event etc etc Yup. It's a commit in git, but yes, there's a command and it's store in the git history as an event.
- wiml 3y ago
  Git doesn't really. It does have a (more or less) immutable, append-only model, but the things stored in git are state snapshots, not events. The UI does tend to present it as if it were events but that's just for convenience.
rswail 3y ago
I've implemented a system that uses event sourcing at "scale" (YMMV at what is "scale"). 1. Data storage is not "10x bigger", it depends on how long you maintain the event stream in the past. Think about a bank account. The "monthly opening balance" is effectively a snapshot in time of all of the previous events for that account. So you can archive events to secondary storage as needed. 2. You don't "fix fuckups by manual database edits/scripts". You make sure that for every business level event, you have a reversal event defined as well. A fuckup is "reverse the affected events, then replay them with corrections". It requires the design at the start to understand that this needs to be dealt with. The benefits: * Each entity has its own state and events, it's self contained and easily mockable and testable on its own. The events are idempotent. * Dependencies between entities is entirely driven by the events, so they are fully decoupled and can be deployed, blue/green, sharded, scaled etc independently. * The events are all that is needed to recreate a scenario or failure, so can be saved and replayed as needed. * Deployment of new entities and/or blue/green and/or scaling becomes trivial. Cons: * If there's an error in production, you have to apply reversal events and then replay the previous events to "fix" a problem. There isn't a "main DB" as such, so there's nothing to change there, but anything that takes the stream of events and turns it into RDBMS (for reporting etc) needs to also understand the reversals and replacement events. This can lead to confusion when a report generated on day X is reversed/redone and now it is "different" to what the report showed at the time. However, this can be covered by actually dealing with the reversals as first-level events in their own right. * When you reverse/replay events for an entity, you need to be aware of side effects on other entities. For example, if you reverse/replay an order, you need to ensure that you don't trigger another round of payment processing. That means adding logic to the payment entity to ensure that it "knows" what order it got payment for so that it doesn't repeat. * Event sourcing is not about "structured data storage", they're entirely about business logic. The events are business events. Programmers seem to have trouble dealing with the fact that the low level storage and processing are independent of the business events. * You need to have versioning of event schemas so that you can add/remove attributes of the event type over time. Your processing has to accomodate the versioning (semver is your friend). * If you have your event processing separate from your query processing, you cannot rely on the query when processing events for the entity itself. If you have multi-threaded and or scaled processing that deals with the entity at the same time, you need to have some form of internal caching/locking/vector-clocks to ensure that you don't have two processes updating the entity from two different events simultaneously. So it has distinct advantages when in production and in the CI/CD deployment. It has additional complexity at the architecture and high level design phase and it needs some standardized libraries and infrastructure to support the necessary features (primarily the internal synchronization of entity/event processing between threads/processes to ensure monotonic updates of the state of the entity).

bob1029 3y ago

> Even simple things such as fixing a typo require a Command to be stored as an Event etc etc. Fixing fuckups ... Event sourcing is a completely different way of thinking about business information. Of course you are going to run into things that look like "downsides". Consider the desired semantics of event sourcing and how your concerns run directly in the face of these semantics. If you aren't comfortable with what qualitatively seem like "trivial" updates, then I don't think a large enough leap of faith has been made. We like to empathize with the machine but sometimes we take it a bit too far. Event sourcing absolutely works and it works at the most massive scales the best. If you can dedicate an entire wing of your 15th floor office to "enterprise messaging and events", this stuff 5000% works. It seems to be at the small scale that it fails most frequently. At small scale, it's usually just one developer who has enough motivation to see it through, and the moment they shift their focus it falls apart. I saw it work in manufacturing in ways I cannot do justice in HN comment size limits.

turnsout 3y ago
Yes, the event model is the only thing that makes sense. Just wanted to say, I don't think Kent's idea is in conflict with the Event Sourcing pattern, as long as these "Effective/Posting" events are stored append-only. Once you start updating the "Effective" date, you're losing data (a customer might call back and ask what the old Effective date was, to understand a paper bill, and you've lost it).
bbkane 3y ago
I would *love* to read a blog post/Google Doc/GitHub Gist/??? about this if you get the time to write one

cratermoon 3y ago

> Does anyone know of a system that successfully did/does event sourcing at scale? The git version control system. > Even simple things such as fixing a typo require a Command to be stored as an Event etc etc Yup. It's a commit in git, but yes, there's a command and it's store in the git history as an event.

wiml 3y ago
Git doesn't really. It does have a (more or less) immutable, append-only model, but the things stored in git are state snapshots, not events. The UI does tend to present it as if it were events but that's just for convenience.

rswail 3y ago

I've implemented a system that uses event sourcing at "scale" (YMMV at what is "scale"). 1. Data storage is not "10x bigger", it depends on how long you maintain the event stream in the past. Think about a bank account. The "monthly opening balance" is effectively a snapshot in time of all of the previous events for that account. So you can archive events to secondary storage as needed. 2. You don't "fix fuckups by manual database edits/scripts". You make sure that for every business level event, you have a reversal event defined as well. A fuckup is "reverse the affected events, then replay them with corrections". It requires the design at the start to understand that this needs to be dealt with. The benefits: * Each entity has its own state and events, it's self contained and easily mockable and testable on its own. The events are idempotent. * Dependencies between entities is entirely driven by the events, so they are fully decoupled and can be deployed, blue/green, sharded, scaled etc independently. * The events are all that is needed to recreate a scenario or failure, so can be saved and replayed as needed. * Deployment of new entities and/or blue/green and/or scaling becomes trivial. Cons: * If there's an error in production, you have to apply reversal events and then replay the previous events to "fix" a problem. There isn't a "main DB" as such, so there's nothing to change there, but anything that takes the stream of events and turns it into RDBMS (for reporting etc) needs to also understand the reversals and replacement events. This can lead to confusion when a report generated on day X is reversed/redone and now it is "different" to what the report showed at the time. However, this can be covered by actually dealing with the reversals as first-level events in their own right. * When you reverse/replay events for an entity, you need to be aware of side effects on other entities. For example, if you reverse/replay an order, you need to ensure that you don't trigger another round of payment processing. That means adding logic to the payment entity to ensure that it "knows" what order it got payment for so that it doesn't repeat. * Event sourcing is not about "structured data storage", they're entirely about business logic. The events are business events. Programmers seem to have trouble dealing with the fact that the low level storage and processing are independent of the business events. * You need to have versioning of event schemas so that you can add/remove attributes of the event type over time. Your processing has to accomodate the versioning (semver is your friend). * If you have your event processing separate from your query processing, you cannot rely on the query when processing events for the entity itself. If you have multi-threaded and or scaled processing that deals with the entity at the same time, you need to have some form of internal caching/locking/vector-clocks to ensure that you don't have two processes updating the entity from two different events simultaneously. So it has distinct advantages when in production and in the CI/CD deployment. It has additional complexity at the architecture and high level design phase and it needs some standardized libraries and infrastructure to support the necessary features (primarily the internal synchronization of entity/event processing between threads/processes to ensure monotonic updates of the state of the entity).