5 ms·

There are a ton of real-world systems that actually do deferred settlement and reconciliation at their distributed edge - for example, in reality ATMs work this

There are a ton of real-world systems that actually do deferred settlement and reconciliation at their distributed edge - for example, in reality ATMs work this way (and credit cards, in a way). These systems should actually be thought of as sources feeding into a distributed log of transactions which are then atomically transacted in a more traditional data store after reconciliation. In systems like this, you must have a well defined system for handling late arrivals, dealing with conflicts, often you need some kind of anti-entropy scheme for correctness station keeping (which people should think about anyway and must people ignore), and so on. These systems are hard to reason about and have many many challenges and many of them that I have personally seen actually are impossible to make robustly secure (a byproduct of having been implemented prior to security being a strong consideration). In these applications the deferred abort problem is dealt with in the settlement and reconciliation phase and these events are themselves just events handled similarly. But this article is blurring the line between underlying transactional data stores where invariants can be guaranteed and the kinds of front-ends that loosen up on 2PC as a requirement. As an observation 2PC is not the problem, the problem is data scope. If the scope of the data necessary for the transaction to operate correctly is narrow enough there is no problem scaling transactional back ends. This gets back to traditional sharding, though, which people don’t like.

kiallmacinnes 8y ago
ATMs are a great example of a system that really don't need a two phase commit protocol. You have an account, it has a balance, and the transaction is a subtraction from the balance. A dumb ATM would read the balance value, subtract, and issue a "set balance to $$$" transaction. A good ATM would check the balance, ensure theres enough money (or overdraft..) for the request, and record a "subtract $40" transaction. If this message gets delayed, oh well, the customer might end up with a negative balance - sucks for the bank if the customer decides to go into hiding - but as the customer typically can't control the delay - it's hard for them to abuse this feature. (I only consider delay here, as I'm sure ATMs make multiple durable copies of their transaction log - making all but the biggest disaster unlikely to prevent them from being able to retrieve the TX's eventually) On the other hand, most systems are nowhere near this "simple". What happens when 3 Github users all simultaneously edit the same Github organisations description? You can't just add/subtract each of the changes. One change has to win, or in other words, two changes need to be rejected. I feel like the authors text really only covers the ATM style use case, a valid use case, but one that's already reasonable to solve without 2 phase commits. Once you are willing to accept & able to handle check+set race conditions, things get much easier :)
- falcor84 8y ago
  I don't quite understand the issue with the GitHub use case. If the operation is just "set name to X", then multiple such operations are trivially serializable and the latest one will win. All prior changes are accepted, performed and immediately overwritten, there's no need for any coordination at all. Or am I missing something?
  - kiallmacinnes 8y ago
    Yes, In my opinion, you're missing the user experience aspect. Two of the three should be rejected - not overridden. - so the users get the feedback they expect, rather than reloading the page and seeing something totally different. And yes - It's a trivial example, one defined by UX - but there are many examples of needing to reject a transaction that can't simply be overridden (or replayed later like simple addition/subtraction) - and I don't see how the authors proposal replaces two phase commit in something like that.
    - amelius 8y ago
      But assume the system updates changes infinitely fast. Then assume user A makes a change, and only after 30 microseconds user B makes a change. Will user A now be confused because their change is overwritten by user B? If so, then the problem has nothing to do with the aforementioned situation and the UX should probably show: "user B is also editing this field" or something like that. The point is: it does not matter if the system is slow and rejects changes; because the effect to the user will be the same as in the "infinitely fast" case.
      - zzzcpan 8y ago
        I think it's fair to assume no collaborative text editing would actually occur on that field at all. And so last write wins is perfectly acceptable strategy here and no coordination is actually necessary, and neither is necessary to inform the user about conflicting edits. For UX it might be useful to observe own edit, but that can be done completely locally with data only eventually propagating farther. Incidentally this is also pretty much the only way to make the edit reliably synchronized with the remote system, because users don't have perfectly reliable computers, perfectly reliable internet connections and won't wait long for anything.
zzzcpan 8y ago
> deferred settlement and reconciliation We call this eventual consistency nowadays.
- PeterCorless 8y ago
  This thread shows more about what people know (or don't know) about the CAP theorem than about two-phase commits.
- foobiekr 8y ago
  Settlement is somewhat different than the EC models you're used to. For one thing, a settlement may never occur and an unwind will be required. At least the EC systems I've dealt with where there application is responsible for resolving the conflict don't really have the concept of doing a chain-of-events unwind. My complaint here is that there actually are two data layers here with completely different semantics and the difficulties of 2PC are meaningfully relevant to the system-of-record layer, not so much the log layer (since enqueue into the log is not really problematic in the real world).
groestl 8y ago
In the context of distributed monetary transactions I think it's worth mentioning CRDTs - for readers of this thread who have not yet heard about them, and who might want to look into the topic if they have a similar problem to solve.
- ben509 8y ago
  Had heard about them, but indirectly. Thanks for putting a name on them! https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type https://en.wikipedia.org/wiki/Conflict-free_replicated_data_...

kiallmacinnes 8y ago

ATMs are a great example of a system that really don't need a two phase commit protocol. You have an account, it has a balance, and the transaction is a subtraction from the balance. A dumb ATM would read the balance value, subtract, and issue a "set balance to $$$" transaction. A good ATM would check the balance, ensure theres enough money (or overdraft..) for the request, and record a "subtract $40" transaction. If this message gets delayed, oh well, the customer might end up with a negative balance - sucks for the bank if the customer decides to go into hiding - but as the customer typically can't control the delay - it's hard for them to abuse this feature. (I only consider delay here, as I'm sure ATMs make multiple durable copies of their transaction log - making all but the biggest disaster unlikely to prevent them from being able to retrieve the TX's eventually) On the other hand, most systems are nowhere near this "simple". What happens when 3 Github users all simultaneously edit the same Github organisations description? You can't just add/subtract each of the changes. One change has to win, or in other words, two changes need to be rejected. I feel like the authors text really only covers the ATM style use case, a valid use case, but one that's already reasonable to solve without 2 phase commits. Once you are willing to accept & able to handle check+set race conditions, things get much easier :)

falcor84 8y ago
I don't quite understand the issue with the GitHub use case. If the operation is just "set name to X", then multiple such operations are trivially serializable and the latest one will win. All prior changes are accepted, performed and immediately overwritten, there's no need for any coordination at all. Or am I missing something?
- kiallmacinnes 8y ago
  Yes, In my opinion, you're missing the user experience aspect. Two of the three should be rejected - not overridden. - so the users get the feedback they expect, rather than reloading the page and seeing something totally different. And yes - It's a trivial example, one defined by UX - but there are many examples of needing to reject a transaction that can't simply be overridden (or replayed later like simple addition/subtraction) - and I don't see how the authors proposal replaces two phase commit in something like that.
  - amelius 8y ago
    But assume the system updates changes infinitely fast. Then assume user A makes a change, and only after 30 microseconds user B makes a change. Will user A now be confused because their change is overwritten by user B? If so, then the problem has nothing to do with the aforementioned situation and the UX should probably show: "user B is also editing this field" or something like that. The point is: it does not matter if the system is slow and rejects changes; because the effect to the user will be the same as in the "infinitely fast" case.
    - zzzcpan 8y ago
      I think it's fair to assume no collaborative text editing would actually occur on that field at all. And so last write wins is perfectly acceptable strategy here and no coordination is actually necessary, and neither is necessary to inform the user about conflicting edits. For UX it might be useful to observe own edit, but that can be done completely locally with data only eventually propagating farther. Incidentally this is also pretty much the only way to make the edit reliably synchronized with the remote system, because users don't have perfectly reliable computers, perfectly reliable internet connections and won't wait long for anything.

zzzcpan 8y ago

> deferred settlement and reconciliation We call this eventual consistency nowadays.

PeterCorless 8y ago
This thread shows more about what people know (or don't know) about the CAP theorem than about two-phase commits.
foobiekr 8y ago
Settlement is somewhat different than the EC models you're used to. For one thing, a settlement may never occur and an unwind will be required. At least the EC systems I've dealt with where there application is responsible for resolving the conflict don't really have the concept of doing a chain-of-events unwind. My complaint here is that there actually are two data layers here with completely different semantics and the difficulties of 2PC are meaningfully relevant to the system-of-record layer, not so much the log layer (since enqueue into the log is not really problematic in the real world).

groestl 8y ago

In the context of distributed monetary transactions I think it's worth mentioning CRDTs - for readers of this thread who have not yet heard about them, and who might want to look into the topic if they have a similar problem to solve.

ben509 8y ago
Had heard about them, but indirectly. Thanks for putting a name on them! https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type https://en.wikipedia.org/wiki/Conflict-free_replicated_data_...