5 ms·

Event Sourcing, CQRS and Micro Services: Real FinTech Example

85 points by fmfamaral 1y ago

throwup238 1y ago
> their MVP was not auditable and thus not compliant with financial regulations and also not scalable (high usage and fault tolerance). There it is. My automatic response to any questions about event sourcing is “if you have to ask, you don’t need it.” This is one of those situations where the explosion in complexity somewhat makes sense: when you need legally enforced auditability. Event sourcing is a really cool architecture that makes theoretical sense but the yak shaving needed to implement it is at least an order of magnitude more than any other design.
- dmoy 1y ago
  Yea I think that's a fair take. If you peer underneath the covers of a lot of financial stuff, and it's effectively double entry accounting. Which is a giant ledger (or ledgers) of events
- chuckadams 1y ago
  I'd call it a trade-off. I've done event sourcing for systems that didn't really need it, and the ability to fix things retroactively by just replaying the log through the corrected pipeline of processors was marvelous. It lends itself to functional programming with effects instead of banging on global state wherever and whenever, and that pays dividends. But there's no free lunch, and pipelines could get complex by needing to track "epochs" and keeping bug-compatible behavior for old messages that would be patched downstream. Since the system just built persisted state at the end, I could checkpoint it at a new state and retire the older epochs. Doubtful that would that pass a rigorous audit, but if that's not one of your requirements, you really can do event sourcing halfway and still reap a lot of the benefits.
- kabes 1y ago
  This! I've recently was hired to fix a company that tried to build their own niche ERP over the last 10 years and they had totally drowned in the confluent kool-aid. There is very very few projects where event sourcing is the best solution.
- marcosdumay 1y ago
  Yes. And that's usually anything that touches money or is limited by it. But you don't need to decide to use it. The people describing the requirements will tell you, insist on it, and threaten you if you don't do it.
doctorpangloss 1y ago
I feel for these guys. The software downturn is steepening. A huge glut of talent with unsustainable comps. And years of toil doing stuff that, in the absence of being paid money for it, nobody would do and no one would care.
nateroling 1y ago
This made me do a double-take. Surely you would never do this, right? It seems to be directly counter to the idea of being able to audit changes: “Event replay: if we want to adjust a past event, for example because it was incorrect, we can just do that and rebuild the app state.”
- javcasas 1y ago
  Yeah, that's a big NO. Events are immutable. If an event is wrong, you post an event with an amendment. Then yes, rebuild the app state.
  - saxenaabhi 1y ago
    Not speaking about their case, but I think some cases a "versioned mutable data store" with a event log that lists updates/inserts makes more sense than an "immutable event log" one like kafka. Consider the update_order_item_quantity event in a classic event sourced systems. It's not possible to guarantee that two waiters dispatching two such events at same time when current quantity is 1 would not cause the quantity to become negative/invalid. If the data store allowed for mutability and produced an event log it's easy: Instead of dispatching the update_order_item_quantity you would update the order document specifying the current version. In the previous example second request would fail since it specified a stale version_id. And you can get the auditability benefits of classic event sourcing system as well because you have versions and an event log. This kind of architecture is trivial to implement with CouchDB and easier to maintain than kafka. Pity it's impossible to find managed hosting for CouchDB outside of IBM.
    - javcasas 1y ago
      Any modern DB with a WAL (write ahead log) is an immutable event system, where the events are the DB primitives (insert, update, delete...). When you construct your own event system you are constructing a DB with your own primitives (deposit, withdraw, transfer, apply monthly interest...). You have to figure out your transaction semantics. For example, how to reject invalid events.
saxenaabhi 1y ago
> INSERT INTO events (account_id, type, amount, timestamp) VALUES (123, 'deposit', 100, NOW()) How would it work if they had to support intra system transfers? So one user balance should be withdrawn and another should get a deposit? That's not possible to do atomically with event sourcing right?
- herval 1y ago
  Do you mean inter-system? Intra-system would mean on the same db, so a simple transaction would do. For inter-system consistency, you’d probably need a reconciliation mechanism or some kind of 2 phase commit
  - saxenaabhi 1y ago
    I mean a bank account sending money to another bank account. That would be 2 events(one withdraw and one deposit)? But if I'm downstream consumer consuming the event log and computing the state from that, if for some reasons I receives only first event the state computed would be invalid and not represent the real state of accounts?
    - manoDev 1y ago
      Nobody relies on atomic transactions to model money transfers in the real world. You can read up on "clearing" and "settlement" processes to get an idea.
    - herval 1y ago
      It could be one event with two database rows inserted, CQRS doesn’t have to map 1:1 to db entries. But in general, these kinds of systems are more complex than 1 or 2 writes, so relying purely on transactions isn’t very common
- javcasas 1y ago
  Create an event type for transfer with amount, source account and destination account.
sfjailbird 1y ago
Traumatic flashbacks to 2017. Glad we moved on from this nonsense. Still dealing with the wreckage, though. Event sourcing is a terrible idea that may be useful for some incredibly niche scenario.
- rawgabbit 1y ago
  What did you find to be the most problematic?
- zigzag312 1y ago
  Data sync is one scenario where it can be useful.
geoffbp 1y ago
The message on his homepage doesn’t make sense right - it should say IT industry? > I am a Software Architect, Ex-Founder & AI enthusiast with over 8 years in the IT.
- wiether 1y ago
  They appear to be ESL and if in Germany they use "IT" the same way we do in France, I can understand why they skipped the "industry" part.
risyachka 1y ago
It kinda sounds like all you needed was a ledger, otherwise didn’t get why would you use CQRS.
- rawgabbit 1y ago
  For me the most important ideas is an immutable ledger and isolating the primary OLTP database from the secondary support services. Reporting, user management, and notifications etc. can be satisfied with a stale copy of the transactional data.
aaronrobinson 1y ago
The answer to Event Sourcing and CQRS is no.
- kccqzy 1y ago
  The answer is rarely so clearly cut. This is a natural extension of the bank system of having both account balance and account transactions. How would you feel if the bank only knows your account balance but not transactions? It has its uses. It's just a bit overused in places where it's unnecessary.
  - aaronrobinson 1y ago
    Fair. I was being flippant.
ff4 1y ago
They already had PostgreSQL (with its strong ACID guarantees) in place, yet the design introduces eventual consistency via MongoDB for reads—without a compelling justification. A DBA could have optimized those PostgreSQL queries to single-digit milliseconds, avoiding the added sync overhead entirely. Instead, it feels like unnecessary complexity was layered onto a proven double-entry ledger approach.
- mrkeen 1y ago
  * Most single-db deployments give up on ACID for performance reasons (see READ_COMMITTED) * Even if you have ACID, it's not sufficient for distributed systems. Its guarantees will keep one node consistent with itself. No transactionality between the customers app and your db. Maybe you're one bank with all the customers, but as soon as you want to talk to other banks, are you really going to share the one ACID instance? Who's the DBA? Currently the state of fintech is 90% of devs being in denial that they're in a distributed system. > proven double-entry ledger approach. Yes. In that language, the ledger is the list of events. If today's devs were around 300 years ago they'd be calling for 'balances' instead of 'ledgers' because they're simpler.
- antonvs 1y ago
  > proven double-entry ledger approach. A double-entry ledger is a combination of a process and a view that was mistaken for a data model centuries ago, and that mistake became embedded. Fundamentally, you’re dealing with a sequence of events. The double-entry ledger is a particular result of processing those events - a view. There are many other useful views. This is well understood in academic accounting. See e.g. https://en.wikipedia.org/wiki/Resources,_Events,_Agents https://en.wikipedia.org/wiki/Resources,_Events,_Agents for an alternative system that doesn’t make the same mistake.
buster 1y ago
Event Sourcing seems like massive overkill for the stated problem. The core requirement is simple: "show account balance at any point in time" for regulatory compliance. What specific audit requirements existed beyond point-in-time balance queries? The author dismisses alternatives as "less business-focused" but doesn't justify why temporal tables or structured audit logs couldn't satisfy the actual compliance need. The performance issues were predictable: 2-5 seconds for balance calculations, requiring complex snapshot strategies to get down to 50-200ms. This entire complexity could have been avoided with a traditional audit trail approach. The business context analogy to accounting ledgers is telling - but accounting systems don't replay every transaction to calculate current balances. They use running totals with audit trails, which is exactly what temporal tables provide. Event Sourcing is elegant from a technical perspective, but here it's solving a problem that simpler, proven approaches handle just fine. The regulatory requirement was for historical balance visibility, not event replay capabilities.
- alecco 1y ago
  Not only overkill, but error-prone. I had to suffer to work on a massive financial system based in serialized Python objects. And expensive as hell. Related: https://vvvvalvalval.github.io/posts/2018-11-12-datomic-event-sourcing-without-the-hassle.html https://vvvvalvalval.github.io/posts/2018-11-12-datomic-even...
- ricardobeat 1y ago
  > They use running totals with audit trails, which is exactly what temporal tables provide. In the author's case, they separate writes and reads into different DBs. The read-optimized DB has aggregated balances stored, not events. This is not materially different, and the trade-offs regarding staleness of data will be mostly the same.
- ealexhudson 1y ago
  I think they needed to be clearer about what the actual requirement was. If the requirement is, "Show the balance _as it was_ at that point in time", this system doesn't fulfil it. They even say so in the article: if something is wrong, throw away the state and re-run the events. That's necessarily different behaviour. To do this requirement, you actually have to audit every enquiry and say what you thought the result was, including the various errors/miscalculations. If the requirement is, "Show the balance as it should have been at that point in time", then it's fine.
coryvirok 1y ago
Not sure why there is so much hate on this thread. I found the post well written, insightful, and pragmatic. Having built systems that process billions of events and displayed results, triggered notifications, etc in real time (not RTOS level, I'm talking 1 or 2 seconds of latency) you absolutely need to separate reads and writes. And if you can trust db replication to be fast and reliable, you can indeed skip distributed locks and stay on the right side of the CAP theorem. Event sourcing is how every write ahead log works. Which powers basically every db. Is the concern on this thread that they preoptimized? I thought they walked through their decision making process pretty clearly.
- ealexhudson 1y ago
  I suspect there is a bit of knee-jerk because so often this pattern is misapplied. I actually quite like the example in the article although I'm basically allergic to CQRS in general. I think your point about write-ahead logging etc is a good one. If you need a decent transactional system, you're probably using a system with some kind of WAL. If you're event sourcing and putting events into something which already implements a WAL, you need to give your head a wobble - why is the same thing being implemented twice? There can be great reasons, but I've seen (a few times) people using a perfectly fine transactional DB of some kind to implement an event store, effectively throwing away all the guarantees of the system underneath.
  - coryvirok 1y ago
    For sure. Event logs in a transactional dbs are weird. I was surprised that they weren't using something like kafka for this.
- mrkeen 1y ago
  > Not sure why there is so much hate on this thread. 1) "Kafka is resume-driven-development" is a meme. 2) Devs are in denial about being in a distributed system, and think that single-threaded thinking (in proximity to a DB that calls itself ACID) leads to correct results in a distributed setting.
torben-friis 1y ago
Semi related question: does anyone have experience introducing proper financial data handling (ledgers or other alternatives) in a fintech _after the fact_ ? As in, fixing things during a scaleup phase when business has been working for a while and the original improvised systems are breaking, but you can’t stop business to repair. Currently undergoing a similar project and would really appreciate any resource thrown my way, both purely technical and/or for interfacing with accounting people with no hybrid roles to bridge the domain gap.
- saxenaabhi 1y ago
  Isn't it simple? Create events from past relational data and add them to the log?
  - torben-friis 1y ago
    Well, yes and no. We’ve got engineers on one side who store mutable relational data (orders, purchases, subscriptions, what have you) and on the other side accountants thinking in terms of accounts (we have such and such in account 705xx). Mapping the two domains is the main issue, and how much the new system should reflect accounting movement of money vs the current engineering model or a completely different in between
    - mrkeen 1y ago
      Your engineers are wrong. You're not going to be able to bridge that gap. Here's a scenario: you've partnered with a credit card provider. They charge some money each month per card, and you pass that onto your customers who use the cards. One day the partner sends you a 'card-cancelled' message. Have you built your system to accept that message unconditionally? Or did your engineers put in defensive code ("fail fast", assertions, status checks, db constraints) so that your system can reject that message? Because that's how we've built our system at work. Our engineers are proud of something called "data integrity" that our almost-ACID (READ_COMMITTED) DB supposedly has. We won't move to events (listening to what actually happened) because we'd be giving up on pretending that our DB guarantees somehow correspond to what's going on in the real world.

85 points by fmfamaral 1y ago

> their MVP was not auditable and thus not compliant with financial regulations and also not scalable (high usage and fault tolerance). There it is. My automatic response to any questions about event sourcing is “if you have to ask, you don’t need it.” This is one of those situations where the explosion in complexity somewhat makes sense: when you need legally enforced auditability. Event sourcing is a really cool architecture that makes theoretical sense but the yak shaving needed to implement it is at least an order of magnitude more than any other design.

dmoy 1y ago
Yea I think that's a fair take. If you peer underneath the covers of a lot of financial stuff, and it's effectively double entry accounting. Which is a giant ledger (or ledgers) of events
chuckadams 1y ago
I'd call it a trade-off. I've done event sourcing for systems that didn't really need it, and the ability to fix things retroactively by just replaying the log through the corrected pipeline of processors was marvelous. It lends itself to functional programming with effects instead of banging on global state wherever and whenever, and that pays dividends. But there's no free lunch, and pipelines could get complex by needing to track "epochs" and keeping bug-compatible behavior for old messages that would be patched downstream. Since the system just built persisted state at the end, I could checkpoint it at a new state and retire the older epochs. Doubtful that would that pass a rigorous audit, but if that's not one of your requirements, you really can do event sourcing halfway and still reap a lot of the benefits.
kabes 1y ago
This! I've recently was hired to fix a company that tried to build their own niche ERP over the last 10 years and they had totally drowned in the confluent kool-aid. There is very very few projects where event sourcing is the best solution.
marcosdumay 1y ago
Yes. And that's usually anything that touches money or is limited by it. But you don't need to decide to use it. The people describing the requirements will tell you, insist on it, and threaten you if you don't do it.

doctorpangloss 1y ago

I feel for these guys. The software downturn is steepening. A huge glut of talent with unsustainable comps. And years of toil doing stuff that, in the absence of being paid money for it, nobody would do and no one would care.

nateroling 1y ago

This made me do a double-take. Surely you would never do this, right? It seems to be directly counter to the idea of being able to audit changes: “Event replay: if we want to adjust a past event, for example because it was incorrect, we can just do that and rebuild the app state.”

javcasas 1y ago
Yeah, that's a big NO. Events are immutable. If an event is wrong, you post an event with an amendment. Then yes, rebuild the app state.
- saxenaabhi 1y ago
  Not speaking about their case, but I think some cases a "versioned mutable data store" with a event log that lists updates/inserts makes more sense than an "immutable event log" one like kafka. Consider the update_order_item_quantity event in a classic event sourced systems. It's not possible to guarantee that two waiters dispatching two such events at same time when current quantity is 1 would not cause the quantity to become negative/invalid. If the data store allowed for mutability and produced an event log it's easy: Instead of dispatching the update_order_item_quantity you would update the order document specifying the current version. In the previous example second request would fail since it specified a stale version_id. And you can get the auditability benefits of classic event sourcing system as well because you have versions and an event log. This kind of architecture is trivial to implement with CouchDB and easier to maintain than kafka. Pity it's impossible to find managed hosting for CouchDB outside of IBM.
  - javcasas 1y ago
    Any modern DB with a WAL (write ahead log) is an immutable event system, where the events are the DB primitives (insert, update, delete...). When you construct your own event system you are constructing a DB with your own primitives (deposit, withdraw, transfer, apply monthly interest...). You have to figure out your transaction semantics. For example, how to reject invalid events.

saxenaabhi 1y ago

> INSERT INTO events (account_id, type, amount, timestamp) VALUES (123, 'deposit', 100, NOW()) How would it work if they had to support intra system transfers? So one user balance should be withdrawn and another should get a deposit? That's not possible to do atomically with event sourcing right?

herval 1y ago
Do you mean inter-system? Intra-system would mean on the same db, so a simple transaction would do. For inter-system consistency, you’d probably need a reconciliation mechanism or some kind of 2 phase commit
- saxenaabhi 1y ago
  I mean a bank account sending money to another bank account. That would be 2 events(one withdraw and one deposit)? But if I'm downstream consumer consuming the event log and computing the state from that, if for some reasons I receives only first event the state computed would be invalid and not represent the real state of accounts?
  - manoDev 1y ago
    Nobody relies on atomic transactions to model money transfers in the real world. You can read up on "clearing" and "settlement" processes to get an idea.
  - herval 1y ago
    It could be one event with two database rows inserted, CQRS doesn’t have to map 1:1 to db entries. But in general, these kinds of systems are more complex than 1 or 2 writes, so relying purely on transactions isn’t very common
javcasas 1y ago
Create an event type for transfer with amount, source account and destination account.

sfjailbird 1y ago

Traumatic flashbacks to 2017. Glad we moved on from this nonsense. Still dealing with the wreckage, though. Event sourcing is a terrible idea that may be useful for some incredibly niche scenario.

rawgabbit 1y ago
What did you find to be the most problematic?
zigzag312 1y ago
Data sync is one scenario where it can be useful.

geoffbp 1y ago

The message on his homepage doesn’t make sense right - it should say IT industry? > I am a Software Architect, Ex-Founder & AI enthusiast with over 8 years in the IT.

wiether 1y ago
They appear to be ESL and if in Germany they use "IT" the same way we do in France, I can understand why they skipped the "industry" part.

risyachka 1y ago

It kinda sounds like all you needed was a ledger, otherwise didn’t get why would you use CQRS.

rawgabbit 1y ago
For me the most important ideas is an immutable ledger and isolating the primary OLTP database from the secondary support services. Reporting, user management, and notifications etc. can be satisfied with a stale copy of the transactional data.

aaronrobinson 1y ago

The answer to Event Sourcing and CQRS is no.

kccqzy 1y ago
The answer is rarely so clearly cut. This is a natural extension of the bank system of having both account balance and account transactions. How would you feel if the bank only knows your account balance but not transactions? It has its uses. It's just a bit overused in places where it's unnecessary.
- aaronrobinson 1y ago
  Fair. I was being flippant.

ff4 1y ago

They already had PostgreSQL (with its strong ACID guarantees) in place, yet the design introduces eventual consistency via MongoDB for reads—without a compelling justification. A DBA could have optimized those PostgreSQL queries to single-digit milliseconds, avoiding the added sync overhead entirely. Instead, it feels like unnecessary complexity was layered onto a proven double-entry ledger approach.

mrkeen 1y ago
* Most single-db deployments give up on ACID for performance reasons (see READ_COMMITTED) * Even if you have ACID, it's not sufficient for distributed systems. Its guarantees will keep one node consistent with itself. No transactionality between the customers app and your db. Maybe you're one bank with all the customers, but as soon as you want to talk to other banks, are you really going to share the one ACID instance? Who's the DBA? Currently the state of fintech is 90% of devs being in denial that they're in a distributed system. > proven double-entry ledger approach. Yes. In that language, the ledger is the list of events. If today's devs were around 300 years ago they'd be calling for 'balances' instead of 'ledgers' because they're simpler.
antonvs 1y ago
> proven double-entry ledger approach. A double-entry ledger is a combination of a process and a view that was mistaken for a data model centuries ago, and that mistake became embedded. Fundamentally, you’re dealing with a sequence of events. The double-entry ledger is a particular result of processing those events - a view. There are many other useful views. This is well understood in academic accounting. See e.g. https://en.wikipedia.org/wiki/Resources,_Events,_Agents https://en.wikipedia.org/wiki/Resources,_Events,_Agents for an alternative system that doesn’t make the same mistake.

buster 1y ago

Event Sourcing seems like massive overkill for the stated problem. The core requirement is simple: "show account balance at any point in time" for regulatory compliance. What specific audit requirements existed beyond point-in-time balance queries? The author dismisses alternatives as "less business-focused" but doesn't justify why temporal tables or structured audit logs couldn't satisfy the actual compliance need. The performance issues were predictable: 2-5 seconds for balance calculations, requiring complex snapshot strategies to get down to 50-200ms. This entire complexity could have been avoided with a traditional audit trail approach. The business context analogy to accounting ledgers is telling - but accounting systems don't replay every transaction to calculate current balances. They use running totals with audit trails, which is exactly what temporal tables provide. Event Sourcing is elegant from a technical perspective, but here it's solving a problem that simpler, proven approaches handle just fine. The regulatory requirement was for historical balance visibility, not event replay capabilities.

alecco 1y ago
Not only overkill, but error-prone. I had to suffer to work on a massive financial system based in serialized Python objects. And expensive as hell. Related: https://vvvvalvalval.github.io/posts/2018-11-12-datomic-event-sourcing-without-the-hassle.html https://vvvvalvalval.github.io/posts/2018-11-12-datomic-even...
ricardobeat 1y ago
> They use running totals with audit trails, which is exactly what temporal tables provide. In the author's case, they separate writes and reads into different DBs. The read-optimized DB has aggregated balances stored, not events. This is not materially different, and the trade-offs regarding staleness of data will be mostly the same.
ealexhudson 1y ago
I think they needed to be clearer about what the actual requirement was. If the requirement is, "Show the balance _as it was_ at that point in time", this system doesn't fulfil it. They even say so in the article: if something is wrong, throw away the state and re-run the events. That's necessarily different behaviour. To do this requirement, you actually have to audit every enquiry and say what you thought the result was, including the various errors/miscalculations. If the requirement is, "Show the balance as it should have been at that point in time", then it's fine.

coryvirok 1y ago

Not sure why there is so much hate on this thread. I found the post well written, insightful, and pragmatic. Having built systems that process billions of events and displayed results, triggered notifications, etc in real time (not RTOS level, I'm talking 1 or 2 seconds of latency) you absolutely need to separate reads and writes. And if you can trust db replication to be fast and reliable, you can indeed skip distributed locks and stay on the right side of the CAP theorem. Event sourcing is how every write ahead log works. Which powers basically every db. Is the concern on this thread that they preoptimized? I thought they walked through their decision making process pretty clearly.

ealexhudson 1y ago
I suspect there is a bit of knee-jerk because so often this pattern is misapplied. I actually quite like the example in the article although I'm basically allergic to CQRS in general. I think your point about write-ahead logging etc is a good one. If you need a decent transactional system, you're probably using a system with some kind of WAL. If you're event sourcing and putting events into something which already implements a WAL, you need to give your head a wobble - why is the same thing being implemented twice? There can be great reasons, but I've seen (a few times) people using a perfectly fine transactional DB of some kind to implement an event store, effectively throwing away all the guarantees of the system underneath.
- coryvirok 1y ago
  For sure. Event logs in a transactional dbs are weird. I was surprised that they weren't using something like kafka for this.
mrkeen 1y ago
> Not sure why there is so much hate on this thread. 1) "Kafka is resume-driven-development" is a meme. 2) Devs are in denial about being in a distributed system, and think that single-threaded thinking (in proximity to a DB that calls itself ACID) leads to correct results in a distributed setting.

torben-friis 1y ago

Semi related question: does anyone have experience introducing proper financial data handling (ledgers or other alternatives) in a fintech _after the fact_ ? As in, fixing things during a scaleup phase when business has been working for a while and the original improvised systems are breaking, but you can’t stop business to repair. Currently undergoing a similar project and would really appreciate any resource thrown my way, both purely technical and/or for interfacing with accounting people with no hybrid roles to bridge the domain gap.

saxenaabhi 1y ago
Isn't it simple? Create events from past relational data and add them to the log?
- torben-friis 1y ago
  Well, yes and no. We’ve got engineers on one side who store mutable relational data (orders, purchases, subscriptions, what have you) and on the other side accountants thinking in terms of accounts (we have such and such in account 705xx). Mapping the two domains is the main issue, and how much the new system should reflect accounting movement of money vs the current engineering model or a completely different in between
  - mrkeen 1y ago
    Your engineers are wrong. You're not going to be able to bridge that gap. Here's a scenario: you've partnered with a credit card provider. They charge some money each month per card, and you pass that onto your customers who use the cards. One day the partner sends you a 'card-cancelled' message. Have you built your system to accept that message unconditionally? Or did your engineers put in defensive code ("fail fast", assertions, status checks, db constraints) so that your system can reject that message? Because that's how we've built our system at work. Our engineers are proud of something called "data integrity" that our almost-ACID (READ_COMMITTED) DB supposedly has. We won't move to events (listening to what actually happened) because we'd be giving up on pretending that our DB guarantees somehow correspond to what's going on in the real world.