4 ms·

The only issue I have with microservices is when you're dealing with atomic things. Like in a monolith you'd probably just stick it all in a database transactio

The only issue I have with microservices is when you're dealing with atomic things. Like in a monolith you'd probably just stick it all in a database transaction. But I can't find any good reads about how to deal with this in a distributed fashion. There's always caveats and ultimately the advice "just try not to do it" but at some point you will probably have an atomic action and you don't want to break your existing architecture. Distributed transactions? Two stage commits? Just do it and rely on the fact you have 99.9% uptime and it's _probably_ not going to fail? Anyone else dealt with this headache?

mark254 5y ago
Well, a simple solution is to only have one middleware that uses database connections, and have all other things around it be purely functional. (Though that may mean using their own databases, of course).
- Philip-J-Fry 5y ago
  What do you mean by that? What if I have 2 services, each with their own databases, but the action is supposed to be atomic. E.g. there's a foreign key that links the data. They're both gonna do their own database functions.
Quekid5 5y ago
IME the only sane way to do this is to a) redraw the transaction boundaries, aka. avoid it, or b) don't do it (see below), c) idempotency -- so you can just retry everything until you succeed. You can do distributed transactions, but it's a huge pain operationally. (b) is not always as far fetched as you might think. The vast majority of Real World (e.g. inventory, order fulfilment generally, etc.) systems are actually much more like Eventual Consistency in practice, so you don't gain anything by being transactional when your Source of Truth is a real world thing. Inventory is a great example. It doesn't matter if your DB says that an item is in stock if there simply isn't an item on the shelf when it comes time to fulfill an order for that item. Tracking such things is always imperfect and there already other systems overlaid which can handle the failure case, e.g. refund the order, place on backorder, have a CSR find a replacement item, etc. etc. (Of course you want reasonable approximate values for analytics, logistics, etc.)
dagss 5y ago
The need to do a cross-service atomic operation indicates that you chose the wrong service boundaries in your architecture. And, since it's microservices, it's near impossible to refactor it, while it could have been a simple thing to reorganize some code in a monolith (where it is also a good idea to make sure that DB transactions don't span wildly different parts of the source code, but the refactor to make that happen is easier). This is the big downside of microservices -- not the difficulty of doing atomic operations, but the difficulty of changing the architecture once you realize you drew the wrong service boundaries. Microservices is great as long as you choose the perfect service boundaries when you start. To me, that's like saying you always write bug-free code the first time -- it's not doable in practice for large complex projects -- hence I'm not a fan of microservices...
- Philip-J-Fry 5y ago
  Yes, this sort of explains my situation. A requirement appears down the line that just completely breaks the service boundaries. An example being something like an online gun store, you have a perfect service that handles orders. It's completely isolated and works fine. But now, 2 years later some local government has asked you "whenever someone buys a gun, you need to call into our webservice the moment the order is placed so we know a gun was sold, and you need to successfully do it, or you can't sell the item" Now you've got a situation where you need an atomic operation, place the order and call the web service, or don't do it at all. You could say just place the order, do the web service call asynchronously and then delete the order afterwards. But you might not have that choice depending on what the regulations say. You can't do it before you place the order because what if payment fails? The order service should not have any idea about blocking orders in specific scenarios. And now the architecture has broken down. Do you add this to the order service and break it's single responsibility? Will this be a bigger problem in the future and do you need to completely rearchitect your solution?
  - dagss 5y ago
    I would say this is another problem. If an external call to a web service is involved, then you can NEVER have an atomic call in the first place. One always needs to just have a state machine to navigate these cases. Even with a monolith, what if you have a power-off at the wrong moment? What you are describing here is to me pretty much the job description of a backend programmer to me -- think through and prepare for what happens if power disappears between code line N and code line N+1 in all situations. In your specific example one would probably use a reserve/capture flow with the payment services provider; first get a reservation for the amount, then do the external webservice call, then finally to a capture call. In our code we pretty much always write "I am about to call external webservice" to our database in one DB transaction (as an event), then call the external webservice, and finally if we get a response, write "I am done calling external webservice" as an event. And then there's a background worker that sits and monitors for cases of "about-to-call events without matching completed-events within 5 minutes", and does according required actions to clean up. If a monolith "solves" this problem then I would say the monolith is buggy. A monolith should also be able to always have a sudden power-off without misbehaving. A power-off between line N and N+1 in a monolith is pretty much the same as a call between two microservices failing at the wrong moment. Not a qualitative difference only a quantitive one (in that power-off MAY be more rare than network errors). Where the difference is is in the things that an ACID database allows you to commit atomically (changes to your internal data either all happening or none happening).
xyzzy123 5y ago
Most people give up and do orchestration instead at some point. Fortunately there are not that many things in the world that need to be 100% atomic so you can get away with a lot. For your own microservices you generally have at least the option of "fixing" the problem properly even if it's at great expense. But then you hit external systems and the problem resurfaces. You can go crazy thinking about this stuff, at a certain point most business logic starts to look like connectors keeping different weird databases in sync, often poorly. Pure crud api? Oh that's a database where client is responsible for orchestration (create folder, upload document...) and some operations might be atomic but there are no transactions for you. Also, the atomicity of any particular operation is not actually guaranteed so it could change next week. Sending an email or an SMS? You're committing to a far away "database" but actually you never know if the commit was successful or not. Payments are a weird one. You can do this perfectly good distributed transaction and then it fails months after it succeeded! Travel booking? runs away screaming so many databases. etc.
p_l 5y ago
Well, there are distributed transaction systems you could use, but usually it's a good idea to ensure things happen inside one transaction in one microservice (also leads to better code in general, IMO - keep your transaction scope as small as possible)
cangencer 5y ago
There are actually some patterns to deal with this, such as Saga - I'm actually working on a project (not open-source yet) related to this specific problem. You can reach me at can@hazelcast.com if you want to learn more.
- polygotdomain 5y ago
  Sagas are definitely what OP should look into here. Quite frankly, transactions are a bit of a fools errand beyond a certain scale, yet are still treated with some absolute purity from the days of Cobb. If you have multiple changes that need to be made, the "all or nothing" approach makes it really simple to deal with and manage. Sagas are about recognizing the individual changes that are necessary, and dealing with the success or failure of them at a higher level. This is complicated though as the developer and the business now need to have a specific conversation around what happens if A succeeds and B fails? Does A need to get "rolled back"? Does B need to be retried? Does C need to wait until B succeeds before proceeding? That all brings in a level of complexity and the only answer is to manage and use the appropriate patterns and tools to do so. Wanting to go back to a land where you can just wrap it all in a transaction so that you get one nice boolean indicating success at the end is quite frankly just naïve. The real world doesn't work like that.

mark254 5y ago

Well, a simple solution is to only have one middleware that uses database connections, and have all other things around it be purely functional. (Though that may mean using their own databases, of course).

Philip-J-Fry 5y ago
What do you mean by that? What if I have 2 services, each with their own databases, but the action is supposed to be atomic. E.g. there's a foreign key that links the data. They're both gonna do their own database functions.

Quekid5 5y ago

IME the only sane way to do this is to a) redraw the transaction boundaries, aka. avoid it, or b) don't do it (see below), c) idempotency -- so you can just retry everything until you succeed. You can do distributed transactions, but it's a huge pain operationally. (b) is not always as far fetched as you might think. The vast majority of Real World (e.g. inventory, order fulfilment generally, etc.) systems are actually much more like Eventual Consistency in practice, so you don't gain anything by being transactional when your Source of Truth is a real world thing. Inventory is a great example. It doesn't matter if your DB says that an item is in stock if there simply isn't an item on the shelf when it comes time to fulfill an order for that item. Tracking such things is always imperfect and there already other systems overlaid which can handle the failure case, e.g. refund the order, place on backorder, have a CSR find a replacement item, etc. etc. (Of course you want reasonable approximate values for analytics, logistics, etc.)

dagss 5y ago

The need to do a cross-service atomic operation indicates that you chose the wrong service boundaries in your architecture. And, since it's microservices, it's near impossible to refactor it, while it could have been a simple thing to reorganize some code in a monolith (where it is also a good idea to make sure that DB transactions don't span wildly different parts of the source code, but the refactor to make that happen is easier). This is the big downside of microservices -- not the difficulty of doing atomic operations, but the difficulty of changing the architecture once you realize you drew the wrong service boundaries. Microservices is great as long as you choose the perfect service boundaries when you start. To me, that's like saying you always write bug-free code the first time -- it's not doable in practice for large complex projects -- hence I'm not a fan of microservices...

Philip-J-Fry 5y ago
Yes, this sort of explains my situation. A requirement appears down the line that just completely breaks the service boundaries. An example being something like an online gun store, you have a perfect service that handles orders. It's completely isolated and works fine. But now, 2 years later some local government has asked you "whenever someone buys a gun, you need to call into our webservice the moment the order is placed so we know a gun was sold, and you need to successfully do it, or you can't sell the item" Now you've got a situation where you need an atomic operation, place the order and call the web service, or don't do it at all. You could say just place the order, do the web service call asynchronously and then delete the order afterwards. But you might not have that choice depending on what the regulations say. You can't do it before you place the order because what if payment fails? The order service should not have any idea about blocking orders in specific scenarios. And now the architecture has broken down. Do you add this to the order service and break it's single responsibility? Will this be a bigger problem in the future and do you need to completely rearchitect your solution?
- dagss 5y ago
  I would say this is another problem. If an external call to a web service is involved, then you can NEVER have an atomic call in the first place. One always needs to just have a state machine to navigate these cases. Even with a monolith, what if you have a power-off at the wrong moment? What you are describing here is to me pretty much the job description of a backend programmer to me -- think through and prepare for what happens if power disappears between code line N and code line N+1 in all situations. In your specific example one would probably use a reserve/capture flow with the payment services provider; first get a reservation for the amount, then do the external webservice call, then finally to a capture call. In our code we pretty much always write "I am about to call external webservice" to our database in one DB transaction (as an event), then call the external webservice, and finally if we get a response, write "I am done calling external webservice" as an event. And then there's a background worker that sits and monitors for cases of "about-to-call events without matching completed-events within 5 minutes", and does according required actions to clean up. If a monolith "solves" this problem then I would say the monolith is buggy. A monolith should also be able to always have a sudden power-off without misbehaving. A power-off between line N and N+1 in a monolith is pretty much the same as a call between two microservices failing at the wrong moment. Not a qualitative difference only a quantitive one (in that power-off MAY be more rare than network errors). Where the difference is is in the things that an ACID database allows you to commit atomically (changes to your internal data either all happening or none happening).

xyzzy123 5y ago

Most people give up and do orchestration instead at some point. Fortunately there are not that many things in the world that need to be 100% atomic so you can get away with a lot. For your own microservices you generally have at least the option of "fixing" the problem properly even if it's at great expense. But then you hit external systems and the problem resurfaces. You can go crazy thinking about this stuff, at a certain point most business logic starts to look like connectors keeping different weird databases in sync, often poorly. Pure crud api? Oh that's a database where client is responsible for orchestration (create folder, upload document...) and some operations might be atomic but there are no transactions for you. Also, the atomicity of any particular operation is not actually guaranteed so it could change next week. Sending an email or an SMS? You're committing to a far away "database" but actually you never know if the commit was successful or not. Payments are a weird one. You can do this perfectly good distributed transaction and then it fails months after it succeeded! Travel booking? runs away screaming so many databases. etc.

p_l 5y ago

Well, there are distributed transaction systems you could use, but usually it's a good idea to ensure things happen inside one transaction in one microservice (also leads to better code in general, IMO - keep your transaction scope as small as possible)

cangencer 5y ago

There are actually some patterns to deal with this, such as Saga - I'm actually working on a project (not open-source yet) related to this specific problem. You can reach me at can@hazelcast.com if you want to learn more.

polygotdomain 5y ago
Sagas are definitely what OP should look into here. Quite frankly, transactions are a bit of a fools errand beyond a certain scale, yet are still treated with some absolute purity from the days of Cobb. If you have multiple changes that need to be made, the "all or nothing" approach makes it really simple to deal with and manage. Sagas are about recognizing the individual changes that are necessary, and dealing with the success or failure of them at a higher level. This is complicated though as the developer and the business now need to have a specific conversation around what happens if A succeeds and B fails? Does A need to get "rolled back"? Does B need to be retried? Does C need to wait until B succeeds before proceeding? That all brings in a level of complexity and the only answer is to manage and use the appropriate patterns and tools to do so. Wanting to go back to a land where you can just wrap it all in a transaction so that you get one nice boolean indicating success at the end is quite frankly just naïve. The real world doesn't work like that.