6 ms·

Distributed transactions are not microservices

supermatt 6y ago
Services can talk to other services. i.e. In your example, the order service can speak to the inventory service to change the stock level. The 'API controller' (this is also a service!) can speak to the inventory service to get the current inventory - but it shouldn't be responsible for handling the transactions of other services. I think the premise of using a monolithic service (your API "Controller") to handle transactions that it simply shouldn't be concerned with is the main problem here. i.e the problem is that the transaction should not be distributed, not that it is handled through multiple services. I'm further confused at the reconciliation that seems to occur later in your example. Why is it bypassing the services and writing directly to the service DB itself?
- agreenlake 6y ago
  Yep. The article seems to muddle up and conflate a bunch of poor practices with "microservices." All the "problems" given are really just poor decomposition of the system into non-discrete components. The "audit" solution is a bit kludgy too.
  - hinkley 6y ago
    > poor decomposition of the system into non-discrete components. You can tar an awful lot of microservices with that brush.
  - ratneshsingh35 6y ago
    in the first place, the components should not be decomposition like this, But you can't control everyone , But if It happen and went to production, the only thing you can do it is make it better, that what I have done in many products (using auditor or arbitrator )
abraxas 6y ago
This speaks volumes about how fraught with peril building microservices is. For me the prevailing message of this post is "don't do it at all unless you are willing to absorb this much complexity for the promised benefits".
- ratneshsingh35 6y ago
  yes, thanks for reading
gav 6y ago
I feel like the e-commerce inventory example isn't a great one because the problem is generally solved by avoiding it. Either: Set the inventory amount in your e-commerce system to be less than the actual inventory (which is rarely accurate anyway). This is your safety stock and depends on how fast moving the item is and if it's a close-out that you are really trying to sell to zero. Then just handle exceptions at allocation-time when you're able to commit stock. Or: Avoid a two-phase commit problem by allocating stock at add-to-cart time with a ticketing system that allows a hold to be placed with a timeout. This is a more customer friendly approach that handles stampedes better, such as caused by marketing emails. Either way, inventory management is like banking, aiming to be eventually consistent is a lot more realistic than being always consistent.
- hangonhn 6y ago
  "Either way, inventory management is like banking, aiming to be eventually consistent is a lot more realistic than being always consistent." Yeah I love those basic DB examples of transactions and why it's important using bank accounts when in real life banks are all eventually consistent because they had to solve the problem before distributed transactions were available.
- ratneshsingh35 6y ago
  Thanks for sharing, If you include scale then the situation might be tricky, you may need to have different DBs and endpoints there will be a limit on how much vertically you can scale, with that thing in mind this problem is not avoidable, but the distributed transaction is still avoidable using eventual inconsistency or a batch proces.
dkhenry 6y ago
This is a really tough problem to solve. I know Alibaba and WePay both have made solutions to try and handle it at scale https://github.com/seata/seata https://github.com/seata/seata https://wecode.wepay.com/posts/waltz-a-distributed-write-ahead-log https://wecode.wepay.com/posts/waltz-a-distributed-write-ahe... At the end of the day I don't think anyone should be coding distributed transactions into their app's. If you need use a solution that abstracts them for you. Eventually we will get one into Vitess ( https://vitess.io https://vitess.io ) when we have figured out something general purpose enough to work for lots of workloads
- mindcrime 6y ago
  JBoss Narayana provides a useful implementation of the Saga Pattern that makes it fairly easy to plug that pattern into microservices interactions.
- ratneshsingh35 6y ago
  in one of the situation, we had a similar problem of scaling and consistency not going hand in hand, at that time batch process solves Lot of our problems.
hinkley 6y ago
A classic: Starbucks does not use two-phase commit https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.html https://www.enterpriseintegrationpatterns.com/ramblings/18_s...
- drwiggly 6y ago
  Hmm there are some reasons this isn't directly applicable. This mode of behavior is possible due to relying on payment processors ability to reliably transfer money, and the business's ability to verify that potential. I don't think Starbucks would with if they accepted hand written IOUs. If you can't rely on an actors intent to such a degree, your actions need to also be less costly to be efficient. Correlated Ids and idempotent end points with retry is pretty much a two phase commit. Something eventually checks and deals with exceptions. Accounting for loss and developing ways to operate with it make a process viable.
- ratneshsingh35 6y ago
  thanks for sharing this, good read
mindcrime 6y ago
Saga Pattern. https://blog.couchbase.com/saga-pattern-implement-business-transactions-using-microservices-part/ https://blog.couchbase.com/saga-pattern-implement-business-t... https://microservices.io/patterns/data/saga.html https://microservices.io/patterns/data/saga.html https://developers.redhat.com/blog/2018/10/01/patterns-for-distributed-transactions-within-a-microservices-architecture/ https://developers.redhat.com/blog/2018/10/01/patterns-for-d... https://www.enterpriseintegrationpatterns.com/patterns/conversation/CompensatingAction.html https://www.enterpriseintegrationpatterns.com/patterns/conve... https://en.wikipedia.org/wiki/Compensating_transaction https://en.wikipedia.org/wiki/Compensating_transaction
- jpalomaki 6y ago
  I think these kind of patterns add huge amount of complexity to the system. Also properly testing them will be quite challenging.
  - pc86 6y ago
    Any library/SDK that allows you to implement these patterns should have sufficient testing scaffolding available as well. We use MassTransit for a large, distributed .NET Core + RabbitMQ service layer and unit tests are no more trouble than they usually are with the build in Bus and Consumer test harnesses.
  - mindcrime 6y ago
    There's no question that it adds complexity, but I wouldn't agree that it adds a "huge amount." The Saga Pattern is actually very straightforward. But as with anything, the question is "is this complexity worth the price you pay for it?" For me, given the advantages that microservices offer in many contexts, I find that using the Saga Pattern to maintain consistent state is totally worth it. But that won't be true for everybody in every situation.
  - eweise 6y ago
    If you build your system around events then the saga pattern doesn't take much additional effort.
one2know 6y ago
Management doesn't care. Management doesn't read this or really even understand what a microservice is. They only know their VP told them to use them and they are the current hot stuff.
- pc86 6y ago
  I'll be honest I get pretty sick of these types of "management doesn't care" comments. Not because they're wrong but because they ignore the obvious solution. If someone at VP-level is making low-level tech decisions, GTFO. If your non-technical executive management even wants to know what the low-level tech driving their business is, GTFO. If your manager, Director, VP, execs, etc will not listen to honest, calm "we really don't need ________ because {5 rational, evidence-backed reasons}," GTFO.
  - colinmorelli 6y ago
    This, but also - these arguments tend to be very one-sided. The implication is that engineers would always do the right thing if pesky management just got out of the way. Is that true sometimes? Of course, there are plenty of shit management teams in the world. There are also plenty of engineers that, left undirected, would add unnecessary scope, introduce unnecessary technology, and create different types of problems. If there really was one answer, and the answer was just as simple as "get rid of the whole management team, and you'll have a much better product at the end" then I have to imagine companies would have started doing this already. My experience being on both sides of this coin in my career is that: it's just not that easy.
    - msla 6y ago
      Management shouldn't care about the stack because that isn't management's job. That's the IT Department's job. The scope of the project is management's job, as is the budget; the friction comes when they want to eat elephant on a dormouse budget, which is a good time for an IT staffer to leave if management is intransigent about refusing to understand trade-offs, but it's also the IT Department's job to explain costs and benefits in a way the non-technical can understand. If the CEO is the lead developer and the head of IT and the CFO what signs all them checks, that's obviously different, and in that case everyone should probably understand everything. Otherwise, ask yourself how deep management gets into the minutiae of washing the toilets.
phs318u 6y ago
A long time ago in a galaxy far away, I remember wishing that our “SOA” stack had support for WS-Transaction. Having come from an Oracle DB experience, I couldn’t understand why anyone would willingly go “backwards” and give up the ability to elegantly manage distributed transactions.
- ratneshsingh35 6y ago
  It seems the term scaling changes everything, SOA is good but in many cases scaling the SOA might not solve the problems and you need to pick the path of optimization, and microservices are the possible way to optimize the service, Now if you need transaction (not in all cases) as well then It becomes tricky.
sa46 6y ago
> Don’t try to build two-phase commit, instead go for an arbitrator pattern which essentially supports resiliency, retry, error handling, timeout handling, and rollback. What’s the arbitrator pattern? Google failed me.
- ratneshsingh35 6y ago
  https://link.springer.com/chapter/10.1007/978-3-642-30885-7_23 https://link.springer.com/chapter/10.1007/978-3-642-30885-7_...
EricE 6y ago
Speaking of micro transactions - this was a recent serendipitous discovery that I’m still digesting but thoroughly enjoying: https://platformdesigntoolkit.com/ https://platformdesigntoolkit.com/

Services can talk to other services. i.e. In your example, the order service can speak to the inventory service to change the stock level. The 'API controller' (this is also a service!) can speak to the inventory service to get the current inventory - but it shouldn't be responsible for handling the transactions of other services. I think the premise of using a monolithic service (your API "Controller") to handle transactions that it simply shouldn't be concerned with is the main problem here. i.e the problem is that the transaction should not be distributed, not that it is handled through multiple services. I'm further confused at the reconciliation that seems to occur later in your example. Why is it bypassing the services and writing directly to the service DB itself?

agreenlake 6y ago
Yep. The article seems to muddle up and conflate a bunch of poor practices with "microservices." All the "problems" given are really just poor decomposition of the system into non-discrete components. The "audit" solution is a bit kludgy too.
- hinkley 6y ago
  > poor decomposition of the system into non-discrete components. You can tar an awful lot of microservices with that brush.
- ratneshsingh35 6y ago
  in the first place, the components should not be decomposition like this, But you can't control everyone , But if It happen and went to production, the only thing you can do it is make it better, that what I have done in many products (using auditor or arbitrator )

abraxas 6y ago

This speaks volumes about how fraught with peril building microservices is. For me the prevailing message of this post is "don't do it at all unless you are willing to absorb this much complexity for the promised benefits".

ratneshsingh35 6y ago
yes, thanks for reading

gav 6y ago

I feel like the e-commerce inventory example isn't a great one because the problem is generally solved by avoiding it. Either: Set the inventory amount in your e-commerce system to be less than the actual inventory (which is rarely accurate anyway). This is your safety stock and depends on how fast moving the item is and if it's a close-out that you are really trying to sell to zero. Then just handle exceptions at allocation-time when you're able to commit stock. Or: Avoid a two-phase commit problem by allocating stock at add-to-cart time with a ticketing system that allows a hold to be placed with a timeout. This is a more customer friendly approach that handles stampedes better, such as caused by marketing emails. Either way, inventory management is like banking, aiming to be eventually consistent is a lot more realistic than being always consistent.

hangonhn 6y ago
"Either way, inventory management is like banking, aiming to be eventually consistent is a lot more realistic than being always consistent." Yeah I love those basic DB examples of transactions and why it's important using bank accounts when in real life banks are all eventually consistent because they had to solve the problem before distributed transactions were available.
ratneshsingh35 6y ago
Thanks for sharing, If you include scale then the situation might be tricky, you may need to have different DBs and endpoints there will be a limit on how much vertically you can scale, with that thing in mind this problem is not avoidable, but the distributed transaction is still avoidable using eventual inconsistency or a batch proces.

dkhenry 6y ago

This is a really tough problem to solve. I know Alibaba and WePay both have made solutions to try and handle it at scale https://github.com/seata/seata https://github.com/seata/seata https://wecode.wepay.com/posts/waltz-a-distributed-write-ahead-log https://wecode.wepay.com/posts/waltz-a-distributed-write-ahe... At the end of the day I don't think anyone should be coding distributed transactions into their app's. If you need use a solution that abstracts them for you. Eventually we will get one into Vitess ( https://vitess.io https://vitess.io ) when we have figured out something general purpose enough to work for lots of workloads

mindcrime 6y ago
JBoss Narayana provides a useful implementation of the Saga Pattern that makes it fairly easy to plug that pattern into microservices interactions.
ratneshsingh35 6y ago
in one of the situation, we had a similar problem of scaling and consistency not going hand in hand, at that time batch process solves Lot of our problems.

hinkley 6y ago

A classic: Starbucks does not use two-phase commit https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.html https://www.enterpriseintegrationpatterns.com/ramblings/18_s...

drwiggly 6y ago
Hmm there are some reasons this isn't directly applicable. This mode of behavior is possible due to relying on payment processors ability to reliably transfer money, and the business's ability to verify that potential. I don't think Starbucks would with if they accepted hand written IOUs. If you can't rely on an actors intent to such a degree, your actions need to also be less costly to be efficient. Correlated Ids and idempotent end points with retry is pretty much a two phase commit. Something eventually checks and deals with exceptions. Accounting for loss and developing ways to operate with it make a process viable.
ratneshsingh35 6y ago
thanks for sharing this, good read

mindcrime 6y ago

Saga Pattern. https://blog.couchbase.com/saga-pattern-implement-business-transactions-using-microservices-part/ https://blog.couchbase.com/saga-pattern-implement-business-t... https://microservices.io/patterns/data/saga.html https://microservices.io/patterns/data/saga.html https://developers.redhat.com/blog/2018/10/01/patterns-for-distributed-transactions-within-a-microservices-architecture/ https://developers.redhat.com/blog/2018/10/01/patterns-for-d... https://www.enterpriseintegrationpatterns.com/patterns/conversation/CompensatingAction.html https://www.enterpriseintegrationpatterns.com/patterns/conve... https://en.wikipedia.org/wiki/Compensating_transaction https://en.wikipedia.org/wiki/Compensating_transaction

jpalomaki 6y ago
I think these kind of patterns add huge amount of complexity to the system. Also properly testing them will be quite challenging.
- pc86 6y ago
  Any library/SDK that allows you to implement these patterns should have sufficient testing scaffolding available as well. We use MassTransit for a large, distributed .NET Core + RabbitMQ service layer and unit tests are no more trouble than they usually are with the build in Bus and Consumer test harnesses.
- mindcrime 6y ago
  There's no question that it adds complexity, but I wouldn't agree that it adds a "huge amount." The Saga Pattern is actually very straightforward. But as with anything, the question is "is this complexity worth the price you pay for it?" For me, given the advantages that microservices offer in many contexts, I find that using the Saga Pattern to maintain consistent state is totally worth it. But that won't be true for everybody in every situation.
- eweise 6y ago
  If you build your system around events then the saga pattern doesn't take much additional effort.

one2know 6y ago

Management doesn't care. Management doesn't read this or really even understand what a microservice is. They only know their VP told them to use them and they are the current hot stuff.

pc86 6y ago
I'll be honest I get pretty sick of these types of "management doesn't care" comments. Not because they're wrong but because they ignore the obvious solution. If someone at VP-level is making low-level tech decisions, GTFO. If your non-technical executive management even wants to know what the low-level tech driving their business is, GTFO. If your manager, Director, VP, execs, etc will not listen to honest, calm "we really don't need ________ because {5 rational, evidence-backed reasons}," GTFO.
- colinmorelli 6y ago
  This, but also - these arguments tend to be very one-sided. The implication is that engineers would always do the right thing if pesky management just got out of the way. Is that true sometimes? Of course, there are plenty of shit management teams in the world. There are also plenty of engineers that, left undirected, would add unnecessary scope, introduce unnecessary technology, and create different types of problems. If there really was one answer, and the answer was just as simple as "get rid of the whole management team, and you'll have a much better product at the end" then I have to imagine companies would have started doing this already. My experience being on both sides of this coin in my career is that: it's just not that easy.
  - msla 6y ago
    Management shouldn't care about the stack because that isn't management's job. That's the IT Department's job. The scope of the project is management's job, as is the budget; the friction comes when they want to eat elephant on a dormouse budget, which is a good time for an IT staffer to leave if management is intransigent about refusing to understand trade-offs, but it's also the IT Department's job to explain costs and benefits in a way the non-technical can understand. If the CEO is the lead developer and the head of IT and the CFO what signs all them checks, that's obviously different, and in that case everyone should probably understand everything. Otherwise, ask yourself how deep management gets into the minutiae of washing the toilets.

phs318u 6y ago

A long time ago in a galaxy far away, I remember wishing that our “SOA” stack had support for WS-Transaction. Having come from an Oracle DB experience, I couldn’t understand why anyone would willingly go “backwards” and give up the ability to elegantly manage distributed transactions.

ratneshsingh35 6y ago
It seems the term scaling changes everything, SOA is good but in many cases scaling the SOA might not solve the problems and you need to pick the path of optimization, and microservices are the possible way to optimize the service, Now if you need transaction (not in all cases) as well then It becomes tricky.

sa46 6y ago

> Don’t try to build two-phase commit, instead go for an arbitrator pattern which essentially supports resiliency, retry, error handling, timeout handling, and rollback. What’s the arbitrator pattern? Google failed me.

ratneshsingh35 6y ago
https://link.springer.com/chapter/10.1007/978-3-642-30885-7_23 https://link.springer.com/chapter/10.1007/978-3-642-30885-7_...

EricE 6y ago

Speaking of micro transactions - this was a recent serendipitous discovery that I’m still digesting but thoroughly enjoying: https://platformdesigntoolkit.com/ https://platformdesigntoolkit.com/