4 ms·

The basic idea looks simple when you have a few entities/event and the numbers of rows in the table are reasonable. It seems to me that the overhead and complex

by tmd83 11y ago

The basic idea looks simple when you have a few entities/event and the numbers of rows in the table are reasonable. It seems to me that the overhead and complexity is somewhat manageable when thats the case. But what do you do or how do you manage the complexity when you have 100s of entities you have tables with rows in the 10s of millions and lots of complicated SELECTS and JOINS to do? I'm not saying that it's impossible but how does people manage complexity at that scale and also the performance overhead.

numbsafari 11y ago
With reasonable abstractions. Someone else mentioned DB transaction logs, but you can also look at Kafka and GetEventStore for additional reference points. That said, you don't necessarily want/need to apply ES to every single entity in your model. Typically, you want to apply it to those entities that have complex lifecycles or that are highly dynamic and might involve a lot of complexity. ES actually often makes it much easier to reason about complex lifecycles because you can isolate the transition logic into discrete units of code. So, it's actually tool for managing complexity. In terms of performance overhead, ES usually results in significant write-side performance improvements because you can either remove or dramatically simplify your transaction coordination overhead (increase concurrency, reduced latency). To address read-side performance, people generally use a cached or materialized view that is lazily updated or implemented with the events as the "write-behind" log for the cached view. This read-side detail is why ES is most often associated with CQRS or lambda-style architectures. As an example, in banking scenarios, you would determine "current balance" by adding up all of the past transactions on an account. Rather than doing this computation every single time you want the current balance, you typically settle the account in a nightly job and store the "nightly balance" and then compute the "current balance" by only considering new transactions for the day.
- tmd83 11y ago
  I understand the basic architecture (or so I think) and its again the question of what happens when you have lots of complex entities with complex workflow/significance. For example in banking the transactions is one core thing while the rest of customer information is not so much, same goes for a the shipping example (or at least it looks that way from the outside). In a health scenario on the other hand I can see most information being required being the type that require an ES model. What I gather from Kafka style architecture is the storing/persistent works through the abstraction the processing itself you have to do separately for each type of entity and if you have many of them and your query requirement are complex not sure how to abstract that. I might be missing something. Again the idea of caching, lazy updating, horizontal scaling those resonates with me but I don't see how to write good abstractions that allows to apply that. For most complex entities it seems I have to do this case by case basis and I can't figure out ways to contain/manage that complexity. And while ES adds for write scalability transactional integrity seems to become much more a complicated questions and hard to abstract about, doesn't it?

by tmd83 11y ago

numbsafari 11y ago

With reasonable abstractions. Someone else mentioned DB transaction logs, but you can also look at Kafka and GetEventStore for additional reference points. That said, you don't necessarily want/need to apply ES to every single entity in your model. Typically, you want to apply it to those entities that have complex lifecycles or that are highly dynamic and might involve a lot of complexity. ES actually often makes it much easier to reason about complex lifecycles because you can isolate the transition logic into discrete units of code. So, it's actually tool for managing complexity. In terms of performance overhead, ES usually results in significant write-side performance improvements because you can either remove or dramatically simplify your transaction coordination overhead (increase concurrency, reduced latency). To address read-side performance, people generally use a cached or materialized view that is lazily updated or implemented with the events as the "write-behind" log for the cached view. This read-side detail is why ES is most often associated with CQRS or lambda-style architectures. As an example, in banking scenarios, you would determine "current balance" by adding up all of the past transactions on an account. Rather than doing this computation every single time you want the current balance, you typically settle the account in a nightly job and store the "nightly balance" and then compute the "current balance" by only considering new transactions for the day.

tmd83 11y ago
I understand the basic architecture (or so I think) and its again the question of what happens when you have lots of complex entities with complex workflow/significance. For example in banking the transactions is one core thing while the rest of customer information is not so much, same goes for a the shipping example (or at least it looks that way from the outside). In a health scenario on the other hand I can see most information being required being the type that require an ES model. What I gather from Kafka style architecture is the storing/persistent works through the abstraction the processing itself you have to do separately for each type of entity and if you have many of them and your query requirement are complex not sure how to abstract that. I might be missing something. Again the idea of caching, lazy updating, horizontal scaling those resonates with me but I don't see how to write good abstractions that allows to apply that. For most complex entities it seems I have to do this case by case basis and I can't figure out ways to contain/manage that complexity. And while ES adds for write scalability transactional integrity seems to become much more a complicated questions and hard to abstract about, doesn't it?