8 ms·
Event Sourcing = everything that happens is an event. Save all the events and you can always get to the latest state, as well as what things look like at any ti
by manigandham 8y ago
Event Sourcing = everything that happens is an event. Save all the events and you can always get to the latest state, as well as what things look like at any time in the past. What an "event" is depends on your business domain and the granularity of processing. It's very common in enterprise apps with complex workflows (like payment processing or manufacturing). Good fit for functional programming techniques and makes business logic easy to reason about since everything is in reaction to some event, and usually emits another event.
With global ordering, you can have a single transaction ID that points to a snapshot of the entire system. The actual communication is usually handled by "service bus" style messaging like RabbitMQ that can support ordering, routing, acknowledgements, and retries. Kafka or a RDBMS can also be used but requires frameworks on top.
This concept is used pretty much everywhere. Redux is event sourcing front-end state for React. Most database replication is event sourcing by using a write-ahead log as the stream of mutations to apply. All that being said, I completely with this article that it's the wrong solution for most cases and creates far more problems and limitations than it solves.
- jsmeaton 8y agoWhat about when event structures change? Now you’re having to push versions into your events and keeping every version of your serialisation format. Redux often does not have to keep track of versions, because the event stream is consistent for that session.
- tomnipotent 8y ago> Now you’re having to push versions into your events Most distributed systems already do this, especially if they use Thrift/Protobuf etc. It's par for the course.
- viraptor 8y agoThe protocol version only handles the version of the structure. It does not change when you change the meaning of the field. For example "this is still foo-index, but from now on it's an index in system X, not system Y". (Yeah, bad example, you could change the field name here, but sometimes it's not that clear)
- eropple 8y agoTypically you write an in-place migration for every version, yes. Or you snapshot your working-copy database and archive the event stream up to that point, and you play it back using the appropriate version of the processing code. It kinda sucks and there isn't a great answer. But there are a lot of use cases where there are real benefits to having that log go back to the start.
- dragonwriter 8y ago> What about when event structures change? Now you’re having to push versions into your events and keeping every version of your serialisation format. Sure. Events are like filled out forms, and there is a reason forms that are significant, modified over time, and where determining meaning from historical forms is a expected business need tend be versioned (often by form publication date). If you ever need to reconstruct/audit history (whatever your data model), you need a log of what happened when that faithfully matches what was recorded at the time, and you need definitions of the semantics as to how the aggregation of all that should map to a final state. Event sourcing is a pretty direct reification of that need, and, yes, versioning is part of what is needed to make that work.
- manigandham 8y agoThe most common approach is to add versions to the events. The good thing is that with event sourcing, the exact cutover and lifetimes of these schema versions can be known (and even recorded as events themselves). Downstream apps and consumers that don't need to be compatible with the entire timeline can then migrate code over time and only deal with the latest version. You have to deal with schemas anytime you have distributed communications anyway, but event sourcing provides a framework for explicit communication and this is one area where it can make things easier.
- tumetab1 8y agoFor me an issue, not usually made explicit, is that those benefits seem to require a pretty stable business and a stable application landscape. Why? Because a changing business requires changes to the Events. Since a change in an Event requires that all consumers to be updated, immediately or at the schema version deprecation, the cost of change seems to increase faster in comparison to an application landscape without event sourcing. At the same time, the cost to know who needs to be changed also grows since "any" application can consume any Event. An stable application landscape also seems to be required because if the number of an Event consumer grows quickly the ability to update, and deprecated schemas, seems to be related with the number of Event consumers (which require update).
- taude 8y agoIf your org is anything like mine, mostly things (data) are "additive" onto the existing structure. When you want to deprecate something, you can notify all the consumers like a thirst party would, if you were going to change something enough. But this later happens much rarer for us, though tends to leave traces of technical debt....
- Turbots 8y agoWhen doing versioning, I would suggest you only maintain the latest two versions at any given time. Supporting more than two versions should be avoided and consuming applications should upgrade to the latest version whenever they can. Consumer-driven contract testing is a good way of testing consumers against new versions of producers. Usually, when the producer only adds a field to an event, the consumer side wont break anyway.
- ouro 8y agoI have actually written a book on this. Its not that hard to deal with. Dealing with it through ad-hoc "we will come up with something as we go along" can be a bit of a pain though. It is in fact fairly trivial to handle if some thought is put into it.
- mvc 8y agoThere is no easy way (in any platform as far as I'm aware), of discovering and updating all clients that depend on some data structure if you plan to change the data structure in an incompatible way. Better to "grow" your data structures than "change" them.
- BlackFly 8y ago> Save all the events and you can always get to the latest state This isn't actually true and good event sourcing guides will point out why this isn't true. Event based systems are naturally racey, on a rerun of a stream of events the order in which the events is processed may change and you might therefore get a different result than the first time. For example, if you have 1 item remaining in inventory but two people attempt to purchase it at the same time, that is a natural race and only 1 of them can succeed. If you are operating at the sort of scale where you require this kind of horizontal scaling, you may be in the territory where these sorts of conflicts become common.
- dtech 8y agoThere should be 1 point in the system that said "user x has ordered item a". That then becomes an event. The command from user x and y to order item a might be a race condition to see which command gets processed first, but once it has been turned into an event it's immutable and without race conditions. You should never replay commands.
- bsaul 8y agoNever done event sourcing architecture, but i guess this kind of subtelty is what makes a system a nightmare or a breeze to work with. Which makes me wonder if there are some kind of exercise book to practice « good architectural design » , just like with algorithms.
- BlackFly 8y agoEvent systems do not require deduplication and/or strong ordering on the ingress. You can also solve the problem on the consumption side and there are often reasons to do this. Plus you should be more forgiving of examples. Let us strengthen the example by saying that the inventory system lied and told us there were 2 items when there was only one. So you issue 2 commands, but when the events arrive at the fully automated warehouse, one randomly receives a null pointer exception and auto-heals into a refund, but which one?
- 8y ago