3 ms·

Are you talking about a "distributed commit" ? In general, the best way to architect such a system is to model a single "Approve" operation as an FSM of multipl

Are you talking about a "distributed commit" ? In general, the best way to architect such a system is to model a single "Approve" operation as an FSM of multiple steps. The queue acts as the state-transition table... however all the databases in question (the accounting DB, payment status, etc.) all have the current status of the operation ... Additionally, you have a garbage-collector "sweeper" that periodically checks the database to make sure that the FSM is not violating any time guarantees. For example, if the entire FSM for the "Approve" action should not take any more than an hour (in the most pathological case), the sweeper reverts back all operations that violate this. this takes care of the case that your queue fails.

ryanjshaw 12y ago
Yes I suppose you could call it a distributed commit of a distributed transaction (hence formalised distributed transactions being one solution). And yes, the design you describe is what I have followed in my applications. What I find odd is that I don't see this pattern being talked about, despite the importance in certain domains... if one goes through a couple of articles on highscalability, it never seems to be discussed. Queues are treated as reliable golden sources. Perhaps it's because a lot of the high volume stuff that's discussed out in the open doesn't need to be 100% reliable (e.g. missing a few Facebook Like click events -- which can be reconciled later anyway) or has a natural replay source (e.g. processing application logs). Software dealing with more business critical scenarios (e.g. financial applications) aren't generally discussed out in the open in detail.
- sandGorgon 12y ago
  yes - I know what you mean. Part of the reason is that there has been an entire body of effort into making queues scale and be available. Interestingly, just today I discovered a new queue that I did not know of before, written in Go [1] that keeps everything in memory and tries to fail over. I think by the time you need a distributed commit, you have the staffing required to build a queue that scales and does not go down. So there is less focus on semantics of distributed commit (which needs a lot of complexity on the application side and is hard to reason about), than to build a scalable queue. [1] https://github.com/bitly/nsq https://github.com/bitly/nsq

ryanjshaw 12y ago

Yes I suppose you could call it a distributed commit of a distributed transaction (hence formalised distributed transactions being one solution). And yes, the design you describe is what I have followed in my applications. What I find odd is that I don't see this pattern being talked about, despite the importance in certain domains... if one goes through a couple of articles on highscalability, it never seems to be discussed. Queues are treated as reliable golden sources. Perhaps it's because a lot of the high volume stuff that's discussed out in the open doesn't need to be 100% reliable (e.g. missing a few Facebook Like click events -- which can be reconciled later anyway) or has a natural replay source (e.g. processing application logs). Software dealing with more business critical scenarios (e.g. financial applications) aren't generally discussed out in the open in detail.

sandGorgon 12y ago
yes - I know what you mean. Part of the reason is that there has been an entire body of effort into making queues scale and be available. Interestingly, just today I discovered a new queue that I did not know of before, written in Go [1] that keeps everything in memory and tries to fail over. I think by the time you need a distributed commit, you have the staffing required to build a queue that scales and does not go down. So there is less focus on semantics of distributed commit (which needs a lot of complexity on the application side and is hard to reason about), than to build a scalable queue. [1] https://github.com/bitly/nsq https://github.com/bitly/nsq