5 ms·

I agree that it's hard, however doable and pays benefits if you know what you're doing. I worked on 3 successful implementations for finance sector and we could

I agree that it's hard, however doable and pays benefits if you know what you're doing. I worked on 3 successful implementations for finance sector and we could replay a few million messages per second. Have a look at how we achieved that in LMAX: https://martinfowler.com/articles/lmax.html https://martinfowler.com/articles/lmax.html Sorry to say it, but clearly you must have been doing something wrong or employing event sourcing where it does not belong.

willvarfar 8y ago
Hmm. You are both right? If LMAX fits your problem, and you're fine with the distribution aspects, and you can adopt a homogeneous architecture etc, then it works well. But the way Event Sourcing is normally sold and implemented is you emit events in some components, written in some mix of languages and half of them probably already legacy, into some 'event bus' thing that you adopt and is probably written internally in some other paradigm, and then you consume these events in several different programs all implemented in other languages by other teams at other times, and you probably do all this across the atlantic! I live in a gazillion-events-per-second world, perhaps not as hectic as finance, but sadly bigger events and sadly globally distributed, and sadly running on amazon (which will be orders of magnitude slower overall than if you have dedicated hardware once you're at this scale because of, aha, aws's complete lack of 'mechanical sympathy' ;) ). It sucks. Oh I wish I had an LMAX-shaped problem.
- macca321 8y ago
  That's EDA (Event Driven Architecture) not ES.
sterlind 8y ago
That article is masterwork, thank you. It seems like having a "hub and spokes" architecture is key to getting event sourcing right. LMAX's Business Logic Processor is the only thing sourcing outputs, so you don't hit dependency hell. Also, your business process is literally a process, rather than living on the ledger. It reminds me of well-designed Paxos control systems like Borg, where the journal serves as the ledger and all business logic lives in one scheduler thread.
kilburn 8y ago
> LMAX's in-memory structures are persistent across input events, so if there is an error it's important to not leave that memory in an inconsistent state. However there's no automated rollback facility. As a consequence the LMAX team puts a lot of attention into ensuring the input events are fully valid before doing any mutation of the in-memory persistent state. They have found that testing is a key tool in flushing out these kinds of problems before going into production. I'm sorry, but this is saying "catch your bugs before they reach production" which just isn't feasible on non-critical software development (i.e., most software development). The important part that is left out here is: what happens when one such errors slips in? How do you deal with it after the fact? That being said, your system is impressive and I loved being able to read about it. Please keep up the good work and specially sharing your findings! :)
- fma 8y ago
  I honestly can't imagine bug free software, even in critical software development. Luckily for me I've worked on important apps, but if there's an issue there is time to trace and repair the data...not a system that has 6 million orders per second.
- kjeetgill 8y ago
  Not quite! I don't think that's what they're getting at. The idea is this: Say you have a record A with fields f1, f2, f3. When an even comes in you run a function F with steps s1, s2, s3 each of which may modify a field of record A. Here's the issue, if s3 fails (due to "invalid input"), the modifications to A from s1 and s2 are incorrect and A is now corrupt. There are a bunch of ways to handle this but the one described here is to avoid touching data that persists between requests until you're at a stage where nothing can fail anymore.
  - kilburn 8y ago
    > until you're at a stage where nothing can fail anymore. ... and then there's a NullPointerException because you forgot to check something that could indeed fail (i.e.: you have a bug). In other words: they advise you to not have bugs in that part of the codebase, which was precisely my objection.

willvarfar 8y ago

Hmm. You are both right? If LMAX fits your problem, and you're fine with the distribution aspects, and you can adopt a homogeneous architecture etc, then it works well. But the way Event Sourcing is normally sold and implemented is you emit events in some components, written in some mix of languages and half of them probably already legacy, into some 'event bus' thing that you adopt and is probably written internally in some other paradigm, and then you consume these events in several different programs all implemented in other languages by other teams at other times, and you probably do all this across the atlantic! I live in a gazillion-events-per-second world, perhaps not as hectic as finance, but sadly bigger events and sadly globally distributed, and sadly running on amazon (which will be orders of magnitude slower overall than if you have dedicated hardware once you're at this scale because of, aha, aws's complete lack of 'mechanical sympathy' ;) ). It sucks. Oh I wish I had an LMAX-shaped problem.

macca321 8y ago
That's EDA (Event Driven Architecture) not ES.

sterlind 8y ago

That article is masterwork, thank you. It seems like having a "hub and spokes" architecture is key to getting event sourcing right. LMAX's Business Logic Processor is the only thing sourcing outputs, so you don't hit dependency hell. Also, your business process is literally a process, rather than living on the ledger. It reminds me of well-designed Paxos control systems like Borg, where the journal serves as the ledger and all business logic lives in one scheduler thread.

kilburn 8y ago

> LMAX's in-memory structures are persistent across input events, so if there is an error it's important to not leave that memory in an inconsistent state. However there's no automated rollback facility. As a consequence the LMAX team puts a lot of attention into ensuring the input events are fully valid before doing any mutation of the in-memory persistent state. They have found that testing is a key tool in flushing out these kinds of problems before going into production. I'm sorry, but this is saying "catch your bugs before they reach production" which just isn't feasible on non-critical software development (i.e., most software development). The important part that is left out here is: what happens when one such errors slips in? How do you deal with it after the fact? That being said, your system is impressive and I loved being able to read about it. Please keep up the good work and specially sharing your findings! :)

fma 8y ago
I honestly can't imagine bug free software, even in critical software development. Luckily for me I've worked on important apps, but if there's an issue there is time to trace and repair the data...not a system that has 6 million orders per second.
kjeetgill 8y ago
Not quite! I don't think that's what they're getting at. The idea is this: Say you have a record A with fields f1, f2, f3. When an even comes in you run a function F with steps s1, s2, s3 each of which may modify a field of record A. Here's the issue, if s3 fails (due to "invalid input"), the modifications to A from s1 and s2 are incorrect and A is now corrupt. There are a bunch of ways to handle this but the one described here is to avoid touching data that persists between requests until you're at a stage where nothing can fail anymore.
- kilburn 8y ago
  > until you're at a stage where nothing can fail anymore. ... and then there's a NullPointerException because you forgot to check something that could indeed fail (i.e.: you have a bug). In other words: they advise you to not have bugs in that part of the codebase, which was precisely my objection.