7 ms·

Transactions in a Microservice World

75 points by kiyanwang 3y ago

numbsafari 3y ago
tl;dr The "Classic Funds Transfer" diagram at the top is the signal that you don't need to read this article and the author probably shouldn't have written it.
- gomezjdaniel 3y ago
  Why?
  - mrkeen 3y ago
    Even though ACID-compliant databases are very trustworthy, it's still bad practice to mutate amounts of money in-place, even if you trust the database not to duplicate or destroy money. "Accountants shouldn't use erasers" and all that. If you're dealing with money, you should at a minimum have some kind of ledger, and derive the current state from that. Calling an SQL transaction "Classic funds transfer" is a bit of a disservice to all the banking that happened before databases. Mesopotamians were recording transactions on clay tablets 5000 years ago.
    - numbsafari 3y ago
      Thank you. Using incorrect and invalid models of real world systems and using them as exemplars of technical concepts, even as a short hand, is a sign of a not well written document. It gets worse, because the "level up" to distributed transactions builds on the model by trying to show two banks engaged in a distributed transaction using 2PC to handle a funds transfer. This doesn't exist. It's not a thing. It never has been. There's nothing "classical" about this example. It exists in the same canon as "a manager is a person. an employee is a person..." examples to explain data modeling. Using these "examples" leads readers (learners) to walk away with incorrect understanding that they will likely spend a ton of time having to unlearn and overcome.
    - hinkley 3y ago
      I've been job hunting lately and one of the positions was for someone trying to disrupt Visa. It took me years of chats with billing folks and dealing with declined purchases to piece together what I know about how credit cards actually work, and it's just the tip of the iceberg. Good luck with your Chesterton's Fence adventure boys. You're gonna need it.
- deleted 3y ago
  [deleted]
- hinkley 3y ago
  The author looks like enough of a grey-hair that he should remember this paper when it first came out. Your Coffee Shop Doesn't Use Two-Phase Commit: https://www.enterpriseintegrationpatterns.com/docs/IEEE_Software_Design_2PC.pdf https://www.enterpriseintegrationpatterns.com/docs/IEEE_Soft... This version was published the same year the term 'microservice' was coined, but the author blogged about it the previous fall.
  - numbsafari 3y ago
    Now _there's_ a classic.
    - yathaid 3y ago
      Are you two annoyed that the author wrote this article when "Your coffee shop ..." exists?
      - hinkley 3y ago
        I’m annoyed it was written 18 years later by someone who should know better. Actually I’m annoyed that a course in distributed computing is an elective at good schools and may not even exist at others. I like concurrency, but I’ve been surprised over and over again to find nobody else can or will step up and it’s because I’m the only one who was classically trained. It’s just ridiculous at this point.
simonw 3y ago
I kind of feel like the moment you're trying to run a transaction across data stored in multiple separate microservices, you're in trouble. "Can entirely encapsulate a transaction" is a good rule of thumb for designing your microservice boundaries. Once that transaction has completed within the microservice, you can have other services then reliably carry out follow-on actions (maybe via an at-least-once delivery queue such as Kafka).
- numbsafari 3y ago
  This is the correct answer, and it only took three sentences and nary a diagram to state. Well done!
- marcosdumay 3y ago
  I kind of feel that every single person that has had good experiences with microservices actually experienced something that doesn't fit the "micro" description, and often enough not even "service". If the thing can encapsulate an entire transaction, then it's taking care of an entire business vertical. What I do agree, it's one of the best ways to partition your services (just after not doing it). But calling this a "microservice" only serves to confuse people.
  - JamesBarney 3y ago
    Yeah can't we just have services without making them tiny. Just because I don't think having 7 services is an appropriate architecture for a team of 3 doesn't mean I don't think it's an appropriate architecture for a team of 100.
  - tsss 3y ago
    It has always been that except for the constant onslaught of misinformed idiots who have no clue about architecture (especially here on hackernews) trying to convince people otherwise.
  - xwolfi 3y ago
    Or get the project approved. Sometimes you dont work with reasonable people: they want micro services, they cannot forgive you for doing one and then not having transactional correctness. So you have to call your business service a micro service, a small lie that saves a lot of pain and money. It's like finishing a migration by renaming your initial project with the new name of the target project: everyone is happy, the software is still working, the various political stakeholders now feel the warm embrace of ownership from the old bad domain to their turf, and you saved years of reinventing the wheel to end up in the same place.
exabrial 3y ago
If you are doing transactions across Microservices you have done "Microservices" incorrectly. Divide your microservices along business, not technical, concerns. While Microservices themselves can/should absolutely use transactions internally, they should be coarse grained actions. A call into a microservice should either fail and complete, but never leave a system in a partial state.
- wg0 3y ago
  And... How that is possible? When you want multiple atomic operations from multiple Microservices to fail altogether or not because there's a workflow that needs those steps?
  - foobazgt 3y ago
    There are all sorts of examples of this in the real world, because it's often not feasible to group everything you'd like into a single transaction. As a trivial example, consider any interaction with a third party. Once you're into that territory, you're talking logs, asynchronous retry queues, compensating operations, etc. Sometimes the surface area of your company's product gets so large that, in order to scale, you interact with other teams as if they were 3rd parties.
    - wg0 3y ago
      We are not talking about third parties. Microservices are always first party, all that you own or some other team within your organisation owns even if they get deployed onto different clouds or something. Easier would be to admit that the microservices idea was the biggest fad in the history of computing and milked by consultancies to no end while compounding the complexity of software development to unprecedented levels, needlessly.
- rqtwteye 3y ago
  Is there a process for figuring out how to do microservices “correctly” by your definition? It seems incredibly difficult to stay correct while requirements are being added.
baz00 3y ago
How this usually turns out... 1. We got into this microservices fad and built this really granular service architecture which we're so proud of but is actually a pile of steaming shit and we're in denial. 2. We forgot to consider that a service boundary is a transaction and/or failure domain and now we have this distributed operation that has to be transactional because it's crapping itself 50 times a day and clients are complaining that something is inconsistent somewhere. 3. Read Martin Fowler's blog and this blog article while having a post-work shit in the bathroom. 4. Come out with a half baked understanding of distributed transactions which is good enough for your ego to think is a really good business outcome. 5. Design another fucking microservice and correlation system for transaction registration. 6. Hand it over to some junior and senior developers who half-ass the whole thing because they don't understand your inane ramblings and have no idea how to test it properly. 7. Throw it into production and shout SUCCESS as loud as possible over the sounds of creaking, grinding and explosions. How this should have turned out... 1. Actually realise that you're only doing about 5 million transactions a day and that's hardly anything. 2. Stuff the whole shebang behind a monolith with a single call, single transaction and in one storage and failure domain. 3. Pay for a slightly fatter DB instance. 4. Go back to developing things with a better ROI. Apologies for the bitterness but it is realistic.
- xnx 3y ago
  A huge amount of inappropriate technology is used because people have incorrect understanding of what scale is.
- exabrial 3y ago
  step 8: move to the next startup for a +$15k salary bump before your code ever sees heavy prod usage. Mark your resume as "expert in web scale".
  - baz00 3y ago
    So so so so true. I am haunted by the ghosts of so many of those bastards. The place is like "oh no they left - quick hire another charlatan".
latchkey 3y ago
Back in 2005, we ran the worlds largest hardcore porn site (kink) on JBoss 4, jgroups with multicast, EJB3 and we did all of this on 3 servers, but only needed one and used the rest of for redundancy on fully automated rolling deploys. They weren't even very special servers... just some Dell blades. We had full transaction isolation and it just ran thousands of connections and transactions per second, gloriously, without any issues. We had a micro currency (kinks), first 1080p pay per minute live viewing, and a whole slew of happy customers with .gov/.mil email addresses. Kids these days with their 'microservices'... psssschhhaaa...
- networkchad 3y ago
  [dead]
- monero-xmr 3y ago
  This times 1000. Devs way over estimate their workloads and way under estimate the ability to scale with fairly low effort. Creating distributed spaghetti is still spaghetti - but now your bowls are spread all over the map.
- anonzzzies 3y ago
  Yep. But I applaud it; let my competitors please adopt AWS, k8s and microservices. We run our monoliths on a few $/mo VPSs while our competitors run 1000-100000s $ per month (literally met the CTO of a UK competitor a few weeks ago at a meetup and he explained their prod + staging for their few clients is now up to $80k/mo at aws; comparing to run around $2000 in total hosting for all our clients together for staging, prod + test. Of course that competitor is playing with investor money so they don't care). Our way requires vastly less resources; humans & machines, so we run far higher margins (in our industry, pricing is identical for all including the sales discounts that are given), we sleep better and more (monoliths are so much easier/faster to debug and deploy) and when things are down (like last year) in the market, we have stacks of profits while two of the top competitors had to get new funding to not just disappear (both did get the funding). I wish everyone many microservices and cutting edge frontend frameworks!
  - 3y ago
francasso 3y ago
How to do transactions in a microservices world: change world.
sarchertech 3y ago
I wrote a guest blog post on the 2 generals problem that goes into a bit about sagas, which this paper touches on briefly in the compensation section. One of these days I’m going to write a follow up blog post called “You (probably) don’t need sagas.” About how at my current company I was able to remove almost all of the sagas we were using because it turns out they were never needed in the first place. https://haydenjames.io/the-two-generals-problem/ https://haydenjames.io/the-two-generals-problem/
- jlledo 3y ago
  Please do, that sounds interesting! If you choose to do so, could you please share a way to get notified when it comes out?
  - sarchertech 3y ago
    Sure thing! You can sign up for the mail list for my free book here https://www.networksfromscratch.com https://www.networksfromscratch.com I also publish updates about related blog posts.

75 points by kiyanwang 3y ago

tl;dr The "Classic Funds Transfer" diagram at the top is the signal that you don't need to read this article and the author probably shouldn't have written it.

gomezjdaniel 3y ago
Why?
- mrkeen 3y ago
  Even though ACID-compliant databases are very trustworthy, it's still bad practice to mutate amounts of money in-place, even if you trust the database not to duplicate or destroy money. "Accountants shouldn't use erasers" and all that. If you're dealing with money, you should at a minimum have some kind of ledger, and derive the current state from that. Calling an SQL transaction "Classic funds transfer" is a bit of a disservice to all the banking that happened before databases. Mesopotamians were recording transactions on clay tablets 5000 years ago.
  - numbsafari 3y ago
    Thank you. Using incorrect and invalid models of real world systems and using them as exemplars of technical concepts, even as a short hand, is a sign of a not well written document. It gets worse, because the "level up" to distributed transactions builds on the model by trying to show two banks engaged in a distributed transaction using 2PC to handle a funds transfer. This doesn't exist. It's not a thing. It never has been. There's nothing "classical" about this example. It exists in the same canon as "a manager is a person. an employee is a person..." examples to explain data modeling. Using these "examples" leads readers (learners) to walk away with incorrect understanding that they will likely spend a ton of time having to unlearn and overcome.
  - hinkley 3y ago
    I've been job hunting lately and one of the positions was for someone trying to disrupt Visa. It took me years of chats with billing folks and dealing with declined purchases to piece together what I know about how credit cards actually work, and it's just the tip of the iceberg. Good luck with your Chesterton's Fence adventure boys. You're gonna need it.
deleted 3y ago
[deleted]
hinkley 3y ago
The author looks like enough of a grey-hair that he should remember this paper when it first came out. Your Coffee Shop Doesn't Use Two-Phase Commit: https://www.enterpriseintegrationpatterns.com/docs/IEEE_Software_Design_2PC.pdf https://www.enterpriseintegrationpatterns.com/docs/IEEE_Soft... This version was published the same year the term 'microservice' was coined, but the author blogged about it the previous fall.
- numbsafari 3y ago
  Now _there's_ a classic.
  - yathaid 3y ago
    Are you two annoyed that the author wrote this article when "Your coffee shop ..." exists?
    - hinkley 3y ago
      I’m annoyed it was written 18 years later by someone who should know better. Actually I’m annoyed that a course in distributed computing is an elective at good schools and may not even exist at others. I like concurrency, but I’ve been surprised over and over again to find nobody else can or will step up and it’s because I’m the only one who was classically trained. It’s just ridiculous at this point.

simonw 3y ago

I kind of feel like the moment you're trying to run a transaction across data stored in multiple separate microservices, you're in trouble. "Can entirely encapsulate a transaction" is a good rule of thumb for designing your microservice boundaries. Once that transaction has completed within the microservice, you can have other services then reliably carry out follow-on actions (maybe via an at-least-once delivery queue such as Kafka).

numbsafari 3y ago
This is the correct answer, and it only took three sentences and nary a diagram to state. Well done!
marcosdumay 3y ago
I kind of feel that every single person that has had good experiences with microservices actually experienced something that doesn't fit the "micro" description, and often enough not even "service". If the thing can encapsulate an entire transaction, then it's taking care of an entire business vertical. What I do agree, it's one of the best ways to partition your services (just after not doing it). But calling this a "microservice" only serves to confuse people.
- JamesBarney 3y ago
  Yeah can't we just have services without making them tiny. Just because I don't think having 7 services is an appropriate architecture for a team of 3 doesn't mean I don't think it's an appropriate architecture for a team of 100.
- tsss 3y ago
  It has always been that except for the constant onslaught of misinformed idiots who have no clue about architecture (especially here on hackernews) trying to convince people otherwise.
- xwolfi 3y ago
  Or get the project approved. Sometimes you dont work with reasonable people: they want micro services, they cannot forgive you for doing one and then not having transactional correctness. So you have to call your business service a micro service, a small lie that saves a lot of pain and money. It's like finishing a migration by renaming your initial project with the new name of the target project: everyone is happy, the software is still working, the various political stakeholders now feel the warm embrace of ownership from the old bad domain to their turf, and you saved years of reinventing the wheel to end up in the same place.

exabrial 3y ago

If you are doing transactions across Microservices you have done "Microservices" incorrectly. Divide your microservices along business, not technical, concerns. While Microservices themselves can/should absolutely use transactions internally, they should be coarse grained actions. A call into a microservice should either fail and complete, but never leave a system in a partial state.

wg0 3y ago
And... How that is possible? When you want multiple atomic operations from multiple Microservices to fail altogether or not because there's a workflow that needs those steps?
- foobazgt 3y ago
  There are all sorts of examples of this in the real world, because it's often not feasible to group everything you'd like into a single transaction. As a trivial example, consider any interaction with a third party. Once you're into that territory, you're talking logs, asynchronous retry queues, compensating operations, etc. Sometimes the surface area of your company's product gets so large that, in order to scale, you interact with other teams as if they were 3rd parties.
  - wg0 3y ago
    We are not talking about third parties. Microservices are always first party, all that you own or some other team within your organisation owns even if they get deployed onto different clouds or something. Easier would be to admit that the microservices idea was the biggest fad in the history of computing and milked by consultancies to no end while compounding the complexity of software development to unprecedented levels, needlessly.
rqtwteye 3y ago
Is there a process for figuring out how to do microservices “correctly” by your definition? It seems incredibly difficult to stay correct while requirements are being added.

baz00 3y ago

How this usually turns out... 1. We got into this microservices fad and built this really granular service architecture which we're so proud of but is actually a pile of steaming shit and we're in denial. 2. We forgot to consider that a service boundary is a transaction and/or failure domain and now we have this distributed operation that has to be transactional because it's crapping itself 50 times a day and clients are complaining that something is inconsistent somewhere. 3. Read Martin Fowler's blog and this blog article while having a post-work shit in the bathroom. 4. Come out with a half baked understanding of distributed transactions which is good enough for your ego to think is a really good business outcome. 5. Design another fucking microservice and correlation system for transaction registration. 6. Hand it over to some junior and senior developers who half-ass the whole thing because they don't understand your inane ramblings and have no idea how to test it properly. 7. Throw it into production and shout SUCCESS as loud as possible over the sounds of creaking, grinding and explosions. How this should have turned out... 1. Actually realise that you're only doing about 5 million transactions a day and that's hardly anything. 2. Stuff the whole shebang behind a monolith with a single call, single transaction and in one storage and failure domain. 3. Pay for a slightly fatter DB instance. 4. Go back to developing things with a better ROI. Apologies for the bitterness but it is realistic.

xnx 3y ago
A huge amount of inappropriate technology is used because people have incorrect understanding of what scale is.
exabrial 3y ago
step 8: move to the next startup for a +$15k salary bump before your code ever sees heavy prod usage. Mark your resume as "expert in web scale".
- baz00 3y ago
  So so so so true. I am haunted by the ghosts of so many of those bastards. The place is like "oh no they left - quick hire another charlatan".

latchkey 3y ago

Back in 2005, we ran the worlds largest hardcore porn site (kink) on JBoss 4, jgroups with multicast, EJB3 and we did all of this on 3 servers, but only needed one and used the rest of for redundancy on fully automated rolling deploys. They weren't even very special servers... just some Dell blades. We had full transaction isolation and it just ran thousands of connections and transactions per second, gloriously, without any issues. We had a micro currency (kinks), first 1080p pay per minute live viewing, and a whole slew of happy customers with .gov/.mil email addresses. Kids these days with their 'microservices'... psssschhhaaa...

networkchad 3y ago
[dead]
monero-xmr 3y ago
This times 1000. Devs way over estimate their workloads and way under estimate the ability to scale with fairly low effort. Creating distributed spaghetti is still spaghetti - but now your bowls are spread all over the map.
anonzzzies 3y ago
Yep. But I applaud it; let my competitors please adopt AWS, k8s and microservices. We run our monoliths on a few $/mo VPSs while our competitors run 1000-100000s $ per month (literally met the CTO of a UK competitor a few weeks ago at a meetup and he explained their prod + staging for their few clients is now up to $80k/mo at aws; comparing to run around $2000 in total hosting for all our clients together for staging, prod + test. Of course that competitor is playing with investor money so they don't care). Our way requires vastly less resources; humans & machines, so we run far higher margins (in our industry, pricing is identical for all including the sales discounts that are given), we sleep better and more (monoliths are so much easier/faster to debug and deploy) and when things are down (like last year) in the market, we have stacks of profits while two of the top competitors had to get new funding to not just disappear (both did get the funding). I wish everyone many microservices and cutting edge frontend frameworks!
- 3y ago

francasso 3y ago

How to do transactions in a microservices world: change world.

sarchertech 3y ago

I wrote a guest blog post on the 2 generals problem that goes into a bit about sagas, which this paper touches on briefly in the compensation section. One of these days I’m going to write a follow up blog post called “You (probably) don’t need sagas.” About how at my current company I was able to remove almost all of the sagas we were using because it turns out they were never needed in the first place. https://haydenjames.io/the-two-generals-problem/ https://haydenjames.io/the-two-generals-problem/

jlledo 3y ago
Please do, that sounds interesting! If you choose to do so, could you please share a way to get notified when it comes out?
- sarchertech 3y ago
  Sure thing! You can sign up for the mail list for my free book here https://www.networksfromscratch.com https://www.networksfromscratch.com I also publish updates about related blog posts.