7 ms·

Distributed transactions in Go: Read before you try

98 points by roblaszczak 2y ago

revskill 2y ago
You can have your cake and eat it too by allowing replication.
p10jkle 2y ago
This is also a good use case for durable execution, see eg https://restate.dev https://restate.dev
liampulles 2y ago
This does a good job of encapsulating the considerations of an event driven system. I think the author is a little too easily dismissive of sagas though - for starters, an event driven system is also still going to need to deal with compensating actions, its just going to result in a larger set of events that various systems potentially need to handle. The virtue of a saga or some command driven orchestration approach is that the story of what happens when a user does X is plainly visible. The ability to dictate that upfront and figure it out easily later when diagnosing issues cannot be understated.
relistan 2y ago
This is a good summary of building an evented system. Having built and run one that scaled up to 130 services and nearly 60 engineers, I can say this solves a lot of problems. Our implementation was a bit different but in a similar vein. When the company didn’t do well in the market and scaled down, 9 engineers were (are) able to operate almost all of that same system. The decoupling and lack of synchronous dependencies means failures are fairly contained and easy to rectify. Replays can fix almost anything after the fact. Scheduled daily replays prevent drift across the system, helping guarantee that things are consistent… eventually.
- latentsea 2y ago
  I would really hate to join a team of 9 engineers that owned 130 services.
physicsguy 2y ago
> And if your product backlog is full and people who designed the microservices are still around, it’s unlikely to happen. Oh man I feel this
ekidd 2y ago
Let's assume you're not a FAANG, and you don't have a billion customers. If you're gluing microservices together using distributed transactions (or durable event queues plus eventual consistency, or whatever), the odds are good that you've gone far down the wrong path. For many applications, it's easiest to start with a modular monolith talking to a shared database, one that natively supports transactions. When this becomes too expensive to scale, the next step may be sharding your backend. (It depends on whether you have a system where users mostly live in their silos, or where everyone talks to everyone. If your users are siloed, you can shard at almost any scale.) Microservices make sense when they're "natural". A video encoder makes a great microservice. So does a map tile generator. Distributed systems are expensive and complicated, and they kill your team's development velocity. I've built several of them. Sometimes, they turned out to be serious mistakes. As a rule of thumb: 1. Design for 10x your current scale, not 1000x. 10x your scale allows for 3 consecutive years of 100% growth before you need to rebuild. Designing for 1000x your scale usually means you're sacrificing development velocity to cosplay as a FAANG. 2. You will want transactions in places that you didn't expect. 3. If you need transactions between two microservices, strongly consider merging them and having them talk to the same database. Sometimes you'll have no better choice than to use distributed transactions or durable event queues. They are inherent in some problems. But they should be treated as a giant flashing "danger" sign.
- hggigg 2y ago
  I would add that just because you add these things it does not mean you can scale afterwards. All microservices implementations I've seen so far are bolted on top of some existing layer of mud and serve only to make function calls that were inside processes run over the network with added latency and other overheads. The end game is the aggregate latency and cost increases only with no functional scalability improvements. Various engineering leads, happy with what went on their resume, leave and tell everyone how they increased scalability. And persuade another generation of failures.
- xyzzy_plugh 2y ago
  I frequently see folks fail to understand that when the unicorn rocketship spends a month and ten of their hundreds of engineers replacing their sharded mysql from setting ablaze daily due to overwhelming load, it is actually pretty close to the correct time for that work. Sure it may have been stressful, and customers may have been impacted, but it's a good problem to have. Conversely not having that problem maybe doesn't really mean anything at all, but there's a good chance it means you were solving these scaling problems prematurely. It's a balancing act, but putting out the fires before they even begin is often the wrong approach. Often a little fire is good for growth.
pjmlp 2y ago
As usual, don't try to use the network boundary to do what modules already offer in most languages. Distributed systems spaghetti is much worse to deal with.
kunley 2y ago
this is smart, but also: the overall design was so overengineed in the first place..
atombender 2y ago
I find the "forwarder" system here a rather awkward way to bridge the database and Pub/Sub system. A better way to do this, I think, is to ignore the term "transaction," which overloaded with too many concepts (such as transactional isolation), and instead to consider the desired behaviour, namely atomicity: You want two updates to happen together, and (1) if one or both fail you want to retry until they are both successful, and (2) if the two updates cannot both be successfully applied within a certain time limit, they should both be undone, or at least flagged for manual intervention. A solution to both (1) and (2) is to bundle both updates into a single action that you retry. You can execute this with a queue-based system. You don't need an outbox for this, because you don't need to create a "bridge" between the database and the following update. Just use Pub/Sub or whatever to enqueue an "update user and apply discount" action. Using acks and nacks, the Pub/Sub worker system can ensure the action is repeatedly retried until both updates complete as a whole. You can build this from basic components like Redis yourself, or you can use a system meant for this type of execution, such as Temporal. To achieve (2), you extend the action's execution with knowledge about whether it should retry or undo its work. For such a simple action as described above, "undo" means taking away the discount and removing the user points, which are just the opposite of the normal action. A durable execution system such as Temporal can help you do that, too. You simply decide, on error, whether to return a "please retry" error, or roll back the previous steps and return a "permanent failure, don't retry" error. To tie this together with an HTTP API that pretends to be synchronous, have the API handler enqueue the task, then wait for its completion. The completion can be a separate queue keyed by a unique ID, so each API request filters on just that completion event. If you're using Redis, you could create a separate Pub/Sub per request. With Temporal, it's simpler: The API handler just starts a workflow and asks for its result, which is a poll operation. The outbox pattern is better in cases where you simply want to bridge between two data processing systems, but where the consumers aren't known. For example, you want all orders to create a Kafka message. The outbox ensures all database changes are eventually guaranteed to land in Kafka, but doesn't know anything about what happens next in Kafka land, which could be stuff that is managed by a different team within the same company, or stuff related to a completely different part of the app, like billing or ops telemetry. But if your app already knows specifically what should happen (because it's a single app with a known data model), the outbox pattern is unnecessary, I think.
cletus 2y ago
To paraphase [1]: > Some people, when confronted with a problem, think “I know, I'll use micro-services.” Now they have two problems. As soon as I read this example where there's users and orders microservices, you've already made an error (IMHO). What happens when the traffic becomes such an issue that you need to shard your microservices? Now you've got session and load-balancing issues. If you ignore them, you may break the read-your-write guarantee and that's going to create a huge cost to development. It goes like this: can you read uncommitted changes within your transaction or request? Generally the answer should be "yes". But imagine you need to speak to a sharded service, what happens when you hit a service that didn't do the mutation but it isn't committed yet? A sharded data backend will take you as far as you need to go. If it's good enough for Facebook, it's good enough for you. When I worked at FB, I had a project where someone had come in from Netflix and they fell into the trap many people do of trying to reinvent Netflix architecture at Facebook. Even if the Netflix microservices architecture is an objectively good idea (which I honestly have no opinion on, other than having personally never seen a good solution with microservices), that train has sailed. FB has embraced a different architecture so even if it's objectively good, you're going against established practice and changing what any FB SWE is going to expect when they come across your system. FB has a write through in-memory graph database (called TAO) that writes to sharded MySQL backends. You almost never speak to MySQL directly. You don't even really talk to TAO directly most of the time. There's a data modelling framework on top of it (that enforces privacy and a lot of other things; talk to TAO directly and you'll have a lot of explaining to do). Anyway, TAO makes the read-your-write promise and the proposed microservices broke that. This was pointed out from the very beginning, yet they barreled on through. I can understand putting video encoding into a "service" but I tend to view those as "workers" more than a "service". [1]: https://regex.info/blog/2006-09-15/247 https://regex.info/blog/2006-09-15/247
- codethief 2y ago
  > Even if the Netflix microservices architecture is an objectively good idea (which I honestly have no opinion on I have no opinion on that either, but at least this[0] story by ThePrimeagen didn't make it sound all too great. (Watch this classic[1] before for context, unless you already know Wingman, Galactus, etc.) [0]: https://youtu.be/s-vJcOfrvi0?t=319 https://youtu.be/s-vJcOfrvi0?t=319 [1]: https://m.youtube.com/watch?v=y8OnoxKotPQ https://m.youtube.com/watch?v=y8OnoxKotPQ
alphazard 2y ago
The best advice (organizationally) is to just do everything in a single transaction on top of Postgres or MySQL for as long as possible. This produces no cognitive overhead for the developers. Sometimes that doesn't deliver enough performance and you need to involve another datastore (or the same datastore across multiple transactions). At that point eventual consistency is a good strategy, much less complicated than distributed transactions. This adds a significant tax to all of your work though. Now everyone has to think through all the states, and additionally design a background process to drive the eventual consistency. Do you have a process in place to ensure all your developers are getting this right for every feature? Did you answer code review? Are you sure there's always enough time to re-do the implementation, and you'll never be forced to merge inconsistent "good enough" behavior? And the worst option (organizationally) is distributed transactions, which basically means a small group of talented engineers can't work on other things and need to be consulted for every new service and most new features and maintain the clients and server for the thumbs up/down system. If you make it hard to do stuff, then people will either 1. do less stuff, or 2. do the same amount of stuff, but badly.
junto 2y ago
This is giving me bad memories of MSDTC and Microsoft SQL Server here.
kgeist 2y ago
Main source of pain with eventual consistency is lots of customer calls/emails "we did X but nothing happened". I'd also add that you should make it clear to the user that the action may not be instantenous.
- renegade-otter 2y ago
  That's another thing about a single database, if it's well-tuned and your squeel is good - in many cases you don't even need any kind of cache, removing an entire class of bugs.
Scubabear68 2y ago
If I had a nickel for all the clients I’ve seen with micro services everywhere, and 90% of the code is replicating an RDBMS with hand coded in memory joins. What could have been a simple SQL query in a sane architecture becomes N REST calls (possibly nested with others downstream) and manually stitching together results. And that is just the read only case. As the author notes updates add another couple of levels of horror.
- renegade-otter 2y ago
  In the good old days, if you did that, you would rightfully be labeled as an "amateur".
wwarner 2y ago
AGREE! The author's point is very well argued. Beginning a transaction is almost never a good idea. Design your data model so that if two pieces of data must be consistent, they are in the same row, and allow associated rows to be missing, handling nulls in the application. Inserts and updates should operate on a single table, because in the case of failure, nothing changed, and you have a simple error to deal with. In short, as explained in the article, embrace eventual consistency. There was a great post from the Github team about why they didn't allow transactions in their rails app, from around 2013, but I can't find it for the life of me. I realize that you're staring at me in disbelief right now, but this is gospel!
latchkey 2y ago
Back in the early 2000's, I was working for the largest hardcore porn company in the world, serving tons of traffic. We built a cluster of 3 Dell 2950 servers with JBoss4. We were using Hibernate and EJB2 entities, with a MySQL backend. This was all before "cloud" allowed porn on their own systems, so we had to do it ourselves. Once configured correctly and all the multicast networking was set up, distributed 2PC transactions via jgroups worked flawlessly for years. We actually only needed one server for all the traffic, but used 3 for redundancy and rolling updates. ¯\_(ツ)_/¯, kids these days
- ebiester 2y ago
  Different problems have different solutions. You likely mostly had very simple business logic in 90% of your system. If your system is automating systems for a cross-domain sector (think payroll), you're likely to have a large number of developers on a relatively small amount of data and speed is secondary to managing the complexity across teams. Microservices might not be a great solution, and distributed monoliths will always be an anti-pattern, but there are reasons for more complex setups to enable concurrent development.
  - latchkey 2y ago
    Due to the unwillingness for corporations to work with us, we had to develop our own cross-TLD login framework, payments system, affiliate tracker, micro-currency for live pay per minute content, secure image/video serving across multiple CDN's, and a whole ads serving network. It took years to build it all and was massively complicated. The point I was making is that the tooling for all of this has existed for ages. People keep reinventing it. Nothing wrong with that, but these sorts of blog posts are entertaining to watch history repeat itself and HN to argue over the best way to do things.
misiek08 2y ago
Looks like crypto ad for the library and showing probably worst, most over-engineered method for „solving” transactions in more diverse environment. Eventually consistency is big tradeoff not possible to accept in many payment and stock related areas. Working in company where all described problems exist and were solved the worst way possible I see this article as very misleading. You don’t want the events instead of transactions - if something has to be commited together - you need to reachitect system and that’s it. Of course people who were building this monster for years will block anyone from doing this. Over-engineered AF, because most of the parts where transactions are required could be handled by single database, even SQL and currently are split between dozens of separate Mongo clusters. "Event based consistency" leaves us in state where you can’t restore system to a stable, safe, consistent state. And of course you have a lot more fun in debugging and developing, because you (we, here) can’t test locally anything. Hundreds of mini-clones of prod setup running, wasting resources and always out of sync are ready to see the change and tell you a little more than nothing. Great DevEx…

98 points by roblaszczak 2y ago

revskill 2y ago

You can have your cake and eat it too by allowing replication.

p10jkle 2y ago

This is also a good use case for durable execution, see eg https://restate.dev https://restate.dev

liampulles 2y ago

This does a good job of encapsulating the considerations of an event driven system. I think the author is a little too easily dismissive of sagas though - for starters, an event driven system is also still going to need to deal with compensating actions, its just going to result in a larger set of events that various systems potentially need to handle. The virtue of a saga or some command driven orchestration approach is that the story of what happens when a user does X is plainly visible. The ability to dictate that upfront and figure it out easily later when diagnosing issues cannot be understated.

relistan 2y ago

This is a good summary of building an evented system. Having built and run one that scaled up to 130 services and nearly 60 engineers, I can say this solves a lot of problems. Our implementation was a bit different but in a similar vein. When the company didn’t do well in the market and scaled down, 9 engineers were (are) able to operate almost all of that same system. The decoupling and lack of synchronous dependencies means failures are fairly contained and easy to rectify. Replays can fix almost anything after the fact. Scheduled daily replays prevent drift across the system, helping guarantee that things are consistent… eventually.

latentsea 2y ago
I would really hate to join a team of 9 engineers that owned 130 services.

physicsguy 2y ago

> And if your product backlog is full and people who designed the microservices are still around, it’s unlikely to happen. Oh man I feel this

ekidd 2y ago

Let's assume you're not a FAANG, and you don't have a billion customers. If you're gluing microservices together using distributed transactions (or durable event queues plus eventual consistency, or whatever), the odds are good that you've gone far down the wrong path. For many applications, it's easiest to start with a modular monolith talking to a shared database, one that natively supports transactions. When this becomes too expensive to scale, the next step may be sharding your backend. (It depends on whether you have a system where users mostly live in their silos, or where everyone talks to everyone. If your users are siloed, you can shard at almost any scale.) Microservices make sense when they're "natural". A video encoder makes a great microservice. So does a map tile generator. Distributed systems are expensive and complicated, and they kill your team's development velocity. I've built several of them. Sometimes, they turned out to be serious mistakes. As a rule of thumb: 1. Design for 10x your current scale, not 1000x. 10x your scale allows for 3 consecutive years of 100% growth before you need to rebuild. Designing for 1000x your scale usually means you're sacrificing development velocity to cosplay as a FAANG. 2. You will want transactions in places that you didn't expect. 3. If you need transactions between two microservices, strongly consider merging them and having them talk to the same database. Sometimes you'll have no better choice than to use distributed transactions or durable event queues. They are inherent in some problems. But they should be treated as a giant flashing "danger" sign.

hggigg 2y ago
I would add that just because you add these things it does not mean you can scale afterwards. All microservices implementations I've seen so far are bolted on top of some existing layer of mud and serve only to make function calls that were inside processes run over the network with added latency and other overheads. The end game is the aggregate latency and cost increases only with no functional scalability improvements. Various engineering leads, happy with what went on their resume, leave and tell everyone how they increased scalability. And persuade another generation of failures.
xyzzy_plugh 2y ago
I frequently see folks fail to understand that when the unicorn rocketship spends a month and ten of their hundreds of engineers replacing their sharded mysql from setting ablaze daily due to overwhelming load, it is actually pretty close to the correct time for that work. Sure it may have been stressful, and customers may have been impacted, but it's a good problem to have. Conversely not having that problem maybe doesn't really mean anything at all, but there's a good chance it means you were solving these scaling problems prematurely. It's a balancing act, but putting out the fires before they even begin is often the wrong approach. Often a little fire is good for growth.

pjmlp 2y ago

As usual, don't try to use the network boundary to do what modules already offer in most languages. Distributed systems spaghetti is much worse to deal with.

kunley 2y ago

this is smart, but also: the overall design was so overengineed in the first place..

atombender 2y ago

I find the "forwarder" system here a rather awkward way to bridge the database and Pub/Sub system. A better way to do this, I think, is to ignore the term "transaction," which overloaded with too many concepts (such as transactional isolation), and instead to consider the desired behaviour, namely atomicity: You want two updates to happen together, and (1) if one or both fail you want to retry until they are both successful, and (2) if the two updates cannot both be successfully applied within a certain time limit, they should both be undone, or at least flagged for manual intervention. A solution to both (1) and (2) is to bundle both updates into a single action that you retry. You can execute this with a queue-based system. You don't need an outbox for this, because you don't need to create a "bridge" between the database and the following update. Just use Pub/Sub or whatever to enqueue an "update user and apply discount" action. Using acks and nacks, the Pub/Sub worker system can ensure the action is repeatedly retried until both updates complete as a whole. You can build this from basic components like Redis yourself, or you can use a system meant for this type of execution, such as Temporal. To achieve (2), you extend the action's execution with knowledge about whether it should retry or undo its work. For such a simple action as described above, "undo" means taking away the discount and removing the user points, which are just the opposite of the normal action. A durable execution system such as Temporal can help you do that, too. You simply decide, on error, whether to return a "please retry" error, or roll back the previous steps and return a "permanent failure, don't retry" error. To tie this together with an HTTP API that pretends to be synchronous, have the API handler enqueue the task, then wait for its completion. The completion can be a separate queue keyed by a unique ID, so each API request filters on just that completion event. If you're using Redis, you could create a separate Pub/Sub per request. With Temporal, it's simpler: The API handler just starts a workflow and asks for its result, which is a poll operation. The outbox pattern is better in cases where you simply want to bridge between two data processing systems, but where the consumers aren't known. For example, you want all orders to create a Kafka message. The outbox ensures all database changes are eventually guaranteed to land in Kafka, but doesn't know anything about what happens next in Kafka land, which could be stuff that is managed by a different team within the same company, or stuff related to a completely different part of the app, like billing or ops telemetry. But if your app already knows specifically what should happen (because it's a single app with a known data model), the outbox pattern is unnecessary, I think.

cletus 2y ago

To paraphase [1]: > Some people, when confronted with a problem, think “I know, I'll use micro-services.” Now they have two problems. As soon as I read this example where there's users and orders microservices, you've already made an error (IMHO). What happens when the traffic becomes such an issue that you need to shard your microservices? Now you've got session and load-balancing issues. If you ignore them, you may break the read-your-write guarantee and that's going to create a huge cost to development. It goes like this: can you read uncommitted changes within your transaction or request? Generally the answer should be "yes". But imagine you need to speak to a sharded service, what happens when you hit a service that didn't do the mutation but it isn't committed yet? A sharded data backend will take you as far as you need to go. If it's good enough for Facebook, it's good enough for you. When I worked at FB, I had a project where someone had come in from Netflix and they fell into the trap many people do of trying to reinvent Netflix architecture at Facebook. Even if the Netflix microservices architecture is an objectively good idea (which I honestly have no opinion on, other than having personally never seen a good solution with microservices), that train has sailed. FB has embraced a different architecture so even if it's objectively good, you're going against established practice and changing what any FB SWE is going to expect when they come across your system. FB has a write through in-memory graph database (called TAO) that writes to sharded MySQL backends. You almost never speak to MySQL directly. You don't even really talk to TAO directly most of the time. There's a data modelling framework on top of it (that enforces privacy and a lot of other things; talk to TAO directly and you'll have a lot of explaining to do). Anyway, TAO makes the read-your-write promise and the proposed microservices broke that. This was pointed out from the very beginning, yet they barreled on through. I can understand putting video encoding into a "service" but I tend to view those as "workers" more than a "service". [1]: https://regex.info/blog/2006-09-15/247 https://regex.info/blog/2006-09-15/247

codethief 2y ago
> Even if the Netflix microservices architecture is an objectively good idea (which I honestly have no opinion on I have no opinion on that either, but at least this[0] story by ThePrimeagen didn't make it sound all too great. (Watch this classic[1] before for context, unless you already know Wingman, Galactus, etc.) [0]: https://youtu.be/s-vJcOfrvi0?t=319 https://youtu.be/s-vJcOfrvi0?t=319 [1]: https://m.youtube.com/watch?v=y8OnoxKotPQ https://m.youtube.com/watch?v=y8OnoxKotPQ

alphazard 2y ago

The best advice (organizationally) is to just do everything in a single transaction on top of Postgres or MySQL for as long as possible. This produces no cognitive overhead for the developers. Sometimes that doesn't deliver enough performance and you need to involve another datastore (or the same datastore across multiple transactions). At that point eventual consistency is a good strategy, much less complicated than distributed transactions. This adds a significant tax to all of your work though. Now everyone has to think through all the states, and additionally design a background process to drive the eventual consistency. Do you have a process in place to ensure all your developers are getting this right for every feature? Did you answer code review? Are you sure there's always enough time to re-do the implementation, and you'll never be forced to merge inconsistent "good enough" behavior? And the worst option (organizationally) is distributed transactions, which basically means a small group of talented engineers can't work on other things and need to be consulted for every new service and most new features and maintain the clients and server for the thumbs up/down system. If you make it hard to do stuff, then people will either 1. do less stuff, or 2. do the same amount of stuff, but badly.

junto 2y ago

This is giving me bad memories of MSDTC and Microsoft SQL Server here.

kgeist 2y ago

Main source of pain with eventual consistency is lots of customer calls/emails "we did X but nothing happened". I'd also add that you should make it clear to the user that the action may not be instantenous.

renegade-otter 2y ago
That's another thing about a single database, if it's well-tuned and your squeel is good - in many cases you don't even need any kind of cache, removing an entire class of bugs.

Scubabear68 2y ago

If I had a nickel for all the clients I’ve seen with micro services everywhere, and 90% of the code is replicating an RDBMS with hand coded in memory joins. What could have been a simple SQL query in a sane architecture becomes N REST calls (possibly nested with others downstream) and manually stitching together results. And that is just the read only case. As the author notes updates add another couple of levels of horror.

renegade-otter 2y ago
In the good old days, if you did that, you would rightfully be labeled as an "amateur".

wwarner 2y ago

AGREE! The author's point is very well argued. Beginning a transaction is almost never a good idea. Design your data model so that if two pieces of data must be consistent, they are in the same row, and allow associated rows to be missing, handling nulls in the application. Inserts and updates should operate on a single table, because in the case of failure, nothing changed, and you have a simple error to deal with. In short, as explained in the article, embrace eventual consistency. There was a great post from the Github team about why they didn't allow transactions in their rails app, from around 2013, but I can't find it for the life of me. I realize that you're staring at me in disbelief right now, but this is gospel!

latchkey 2y ago

Back in the early 2000's, I was working for the largest hardcore porn company in the world, serving tons of traffic. We built a cluster of 3 Dell 2950 servers with JBoss4. We were using Hibernate and EJB2 entities, with a MySQL backend. This was all before "cloud" allowed porn on their own systems, so we had to do it ourselves. Once configured correctly and all the multicast networking was set up, distributed 2PC transactions via jgroups worked flawlessly for years. We actually only needed one server for all the traffic, but used 3 for redundancy and rolling updates. ¯\_(ツ)_/¯, kids these days

ebiester 2y ago
Different problems have different solutions. You likely mostly had very simple business logic in 90% of your system. If your system is automating systems for a cross-domain sector (think payroll), you're likely to have a large number of developers on a relatively small amount of data and speed is secondary to managing the complexity across teams. Microservices might not be a great solution, and distributed monoliths will always be an anti-pattern, but there are reasons for more complex setups to enable concurrent development.
- latchkey 2y ago
  Due to the unwillingness for corporations to work with us, we had to develop our own cross-TLD login framework, payments system, affiliate tracker, micro-currency for live pay per minute content, secure image/video serving across multiple CDN's, and a whole ads serving network. It took years to build it all and was massively complicated. The point I was making is that the tooling for all of this has existed for ages. People keep reinventing it. Nothing wrong with that, but these sorts of blog posts are entertaining to watch history repeat itself and HN to argue over the best way to do things.

misiek08 2y ago

Looks like crypto ad for the library and showing probably worst, most over-engineered method for „solving” transactions in more diverse environment. Eventually consistency is big tradeoff not possible to accept in many payment and stock related areas. Working in company where all described problems exist and were solved the worst way possible I see this article as very misleading. You don’t want the events instead of transactions - if something has to be commited together - you need to reachitect system and that’s it. Of course people who were building this monster for years will block anyone from doing this. Over-engineered AF, because most of the parts where transactions are required could be handled by single database, even SQL and currently are split between dozens of separate Mongo clusters. "Event based consistency" leaves us in state where you can’t restore system to a stable, safe, consistent state. And of course you have a lot more fun in debugging and developing, because you (we, here) can’t test locally anything. Hundreds of mini-clones of prod setup running, wasting resources and always out of sync are ready to see the change and tell you a little more than nothing. Great DevEx…