3 ms·

What cannot the client forgive you for not ensuring: I faced this case, we had an ivory tower-designed micro service architecture, the travel insurance team dec

by xwolfi 3y ago

What cannot the client forgive you for not ensuring: I faced this case, we had an ivory tower-designed micro service architecture, the travel insurance team decided to use it to not be fired or humiliated by the architect 2 floors above. They proceed to create 5 services in python aws lambdas: ui proxy, credit card processor, insurance gateway, email service. The flow would be: the user clicks buys, it sends the the ui proxy, which triggers a redirect to a credit card ui, the client pays, the backend sends a request to the insurance gateway, and if all goes well, an email is sent with the policy. Ofc, we didnt receive a policy document from the insurance gateway, we'd generate it ourselves on success, the insurance gateway only saying "ok". The first few weeks, the team missed some failure condition on the insurance gateway, and would charge the client and send a policy pdf by email, but the client was NOT actually insured. It impacted 7 clients a day going to trips thinking they were insured but were NOT. Imagine the face of the business when they realized there was no transactional boundary over the whole process: one micro service could fail and the others could succeed and god knows what happens then. The change was to build one giant procedural thing that does 1 step at a time and finishes once all is good. One service, one transaction, one state that matters. The architect was eventually fired, finally.

seer 3y ago
Hmm, wouldn’t things have ended up the same inconsistent state if you had it all built as a single service in the first place? E.g. you still can miss a failure condition on the insurance side and end up with a committed transaction? Why was the failure missed in your opinion? Was it because of the opaqueness of the services? If you go monolith, you’ll of course have the advantage of having one place that orchestrates everything, the place where you start the transaction and follow up with credit card / insurance / email. But if you are forced to deal with micro services you can roll a poor man’s transaction in the form of a state machine for the whole process backed in the db. Ui proxy receives click, credit card service authorizes money, insurance generates, credit card service settles transaction, email is sent. The good thing about this whole affair is that you can very clearly see where something fails and why, because failure is not a special case, but part of the normal operation of the service. Also gets you the whole split up into micro services thing where different teams can handle different services. Bad thing of course is its massively more complex, so most probably is not justified. Transactions are great, but when they rollback, they rollback everything. Now if you want to actually understand and track what went wrong, you need to log stuff somewhere else, which in itself can also fail.

by xwolfi 3y ago

seer 3y ago

Hmm, wouldn’t things have ended up the same inconsistent state if you had it all built as a single service in the first place? E.g. you still can miss a failure condition on the insurance side and end up with a committed transaction? Why was the failure missed in your opinion? Was it because of the opaqueness of the services? If you go monolith, you’ll of course have the advantage of having one place that orchestrates everything, the place where you start the transaction and follow up with credit card / insurance / email. But if you are forced to deal with micro services you can roll a poor man’s transaction in the form of a state machine for the whole process backed in the db. Ui proxy receives click, credit card service authorizes money, insurance generates, credit card service settles transaction, email is sent. The good thing about this whole affair is that you can very clearly see where something fails and why, because failure is not a special case, but part of the normal operation of the service. Also gets you the whole split up into micro services thing where different teams can handle different services. Bad thing of course is its massively more complex, so most probably is not justified. Transactions are great, but when they rollback, they rollback everything. Now if you want to actually understand and track what went wrong, you need to log stuff somewhere else, which in itself can also fail.