6 ms·

I have been working on payment systems and it seems that in almost all discussions about transactions, people talk about toy versions of bank transactions that

I have been working on payment systems and it seems that in almost all discussions about transactions, people talk about toy versions of bank transactions that have very little to do with what actually happens. You don't even need to talk about credit cards to have multiple kinds of accounts (internal bank accounts for payment settlement etc.), multiple involved systems, batch processes, reconciliation etc. Having a single atomic database transaction is not realistic at all. On the other hand, the toy transaction example might be useful for people to understand basic concepts of transactions.

awesome_dude 9mo ago
The point is to give people who don't realise that they have been dealing with eventual consistency all along, that it's right there, in their lives, and they already understand it. You're right I go into too much detail (maybe I got carried away with the HN audience :-) and you are right that multiple accounts is something else that people generally already understand and demonstrate further eventual consistency principles.
- kukkeliskuu 9mo ago
  I wasn't criticizing you, just making the point that when people talk about toy example bank transactions, they usually want to just introduce the basic understanding. And I think it ok, but I would prefer that they would also mention that REALLY the operations are complex. I modified my comment above that by multiple types of accounts I meant that banks have various accounts for settlements with the other banks etc. even in the common payment case.
  - awesome_dude 9mo ago
    My bad, I didn't mean to sound too upset, but I do get a bit "trigger happy" from time to time
mrkeen 9mo ago
And then they take that toy transaction model and think that they're on ACID when they're not. Are you stepping out of SQL to write application logic? You probably broke ACID. Begin a transaction, read a value (n), do a calculation (n+1), write it back and commit: The DB cannot see that you did (+1). All it knows is that you're trying to write a 6. If someone else wrote a 6 or a 7 in the meantime, then your transaction may have 'meant' (+0) or (-1). Same problem when running on reduced isolation level (which you probably are). If you do two reads in your 'transaction', the first read can be at state 1, and the second read can be at state 2. I think more conversations about the single "fully consistent" db approach should start with it not being fit-for-purpose - even without considering that it can't address soft-modification (which you should recognise as a need immediately whenever someone brings up soft-delete) or two-generals (i.e. consistency with a partner - you and VISA don't live in the same MySql instance, do you? Or to put it in moron words - partitions between your DB and VISA's DB "don't happen often" (they happen always!))
- fishstamp82 9mo ago
  RE: "All it knows is that you're trying to write a 6. If someone else wrote a 6 or a 7 in the meantime, then your transaction may have 'meant' (+0) or (-1)." This is not how it works at all. This is called dirty writes and is by default prevented by ACID compliant databases, no matter the isolation level. The second transaction commit will be rejected by the transaction manager. Even if you start a transaction from your application, it does not change this still.
  - belter 9mo ago
    http://www.bailis.org/blog/when-is-acid-acid-rarely/ http://www.bailis.org/blog/when-is-acid-acid-rarely/
    - fishstamp82 9mo ago
      Postgres as an example is ACID compliant if you want it to be. All those databases that have full serialization possible do utilize RC by default which is enough to prevent dirty writes and was my original point. Thanks for the link still, it was valuable!
arter45 9mo ago
I don't have a lot of payment experience, but AFAIK actual payment systems work in an append-only fashion, which makes concurrency management easier since you're just adding a new row with (timestamp, from, to, value, currency, status) or something similar. However, how can you efficiently check for overdrafts in this model? You'd have to periodically sum up transactions to find the sender's balance and compare it to a known threshold. Is this how things are usually done in your business domain?
- kukkeliskuu 8mo ago
  > Is this how things are usually done in your business domain? I don't know about "usually" and I cannot explain details. But many banks are migrating from batch-based mainframes to real-time systems. Maybe that answers your question about "efficiently".
- mrkeen 9mo ago
  > how can you efficiently check for overdrafts in this model? You already laid the groundwork for this to be done efficiently: "actual payment systems work in an append-only fashion" If you can't alter the past, it's trivial to maintain your rolling sums to compare against. Each new transaction through the system only needs to mutate the source and destination balances of that individual transaction. If you know everyone's balance as of 10 seconds ago, you don't need to consider any of the 5 million transactions that happened before 10 seconds ago. (If your system allowed you to alter the past and edit arbitrary transactions in the past, you could never trust your rolling sums, and you'd be back to summing up everything for every operation.)
  - arter45 9mo ago
    So you're saying each line records the new value of the source and destination balance, rather than just the sum that is being exchanged?
    - mrkeen 9mo ago
      No. At the beginning of time, all your accounts will have their starting value. When the first transaction (from,to,value) happens, you will do one overdraft check, and if it's good, you will do 1 addition and 1 subtraction, and two of the accounts will have a new value. On the millionth transaction, you will do one overdraft check, and if it's good, you will do 1 addition and 1 subtraction, and two of the accounts will have a new value. At no point will you need to do more than one check & one add & one sub per arriving transaction. (The append-only is what allows this: the next state is only ever a single, cheap step from the current state. But if someone insists upon mutating history, the current state is no longer valid because it no longer represents the history that led up to it, so it cannot be used to generate the next state - you need to throw it all away and regenerate the current/next states, starting from 0 and replaying every transaction again.

awesome_dude 9mo ago

The point is to give people who don't realise that they have been dealing with eventual consistency all along, that it's right there, in their lives, and they already understand it. You're right I go into too much detail (maybe I got carried away with the HN audience :-) and you are right that multiple accounts is something else that people generally already understand and demonstrate further eventual consistency principles.

kukkeliskuu 9mo ago
I wasn't criticizing you, just making the point that when people talk about toy example bank transactions, they usually want to just introduce the basic understanding. And I think it ok, but I would prefer that they would also mention that REALLY the operations are complex. I modified my comment above that by multiple types of accounts I meant that banks have various accounts for settlements with the other banks etc. even in the common payment case.
- awesome_dude 9mo ago
  My bad, I didn't mean to sound too upset, but I do get a bit "trigger happy" from time to time

mrkeen 9mo ago

And then they take that toy transaction model and think that they're on ACID when they're not. Are you stepping out of SQL to write application logic? You probably broke ACID. Begin a transaction, read a value (n), do a calculation (n+1), write it back and commit: The DB cannot see that you did (+1). All it knows is that you're trying to write a 6. If someone else wrote a 6 or a 7 in the meantime, then your transaction may have 'meant' (+0) or (-1). Same problem when running on reduced isolation level (which you probably are). If you do two reads in your 'transaction', the first read can be at state 1, and the second read can be at state 2. I think more conversations about the single "fully consistent" db approach should start with it not being fit-for-purpose - even without considering that it can't address soft-modification (which you should recognise as a need immediately whenever someone brings up soft-delete) or two-generals (i.e. consistency with a partner - you and VISA don't live in the same MySql instance, do you? Or to put it in moron words - partitions between your DB and VISA's DB "don't happen often" (they happen always!))

fishstamp82 9mo ago
RE: "All it knows is that you're trying to write a 6. If someone else wrote a 6 or a 7 in the meantime, then your transaction may have 'meant' (+0) or (-1)." This is not how it works at all. This is called dirty writes and is by default prevented by ACID compliant databases, no matter the isolation level. The second transaction commit will be rejected by the transaction manager. Even if you start a transaction from your application, it does not change this still.
- belter 9mo ago
  http://www.bailis.org/blog/when-is-acid-acid-rarely/ http://www.bailis.org/blog/when-is-acid-acid-rarely/
  - fishstamp82 9mo ago
    Postgres as an example is ACID compliant if you want it to be. All those databases that have full serialization possible do utilize RC by default which is enough to prevent dirty writes and was my original point. Thanks for the link still, it was valuable!

arter45 9mo ago

I don't have a lot of payment experience, but AFAIK actual payment systems work in an append-only fashion, which makes concurrency management easier since you're just adding a new row with (timestamp, from, to, value, currency, status) or something similar. However, how can you efficiently check for overdrafts in this model? You'd have to periodically sum up transactions to find the sender's balance and compare it to a known threshold. Is this how things are usually done in your business domain?

kukkeliskuu 8mo ago
> Is this how things are usually done in your business domain? I don't know about "usually" and I cannot explain details. But many banks are migrating from batch-based mainframes to real-time systems. Maybe that answers your question about "efficiently".
mrkeen 9mo ago
> how can you efficiently check for overdrafts in this model? You already laid the groundwork for this to be done efficiently: "actual payment systems work in an append-only fashion" If you can't alter the past, it's trivial to maintain your rolling sums to compare against. Each new transaction through the system only needs to mutate the source and destination balances of that individual transaction. If you know everyone's balance as of 10 seconds ago, you don't need to consider any of the 5 million transactions that happened before 10 seconds ago. (If your system allowed you to alter the past and edit arbitrary transactions in the past, you could never trust your rolling sums, and you'd be back to summing up everything for every operation.)
- arter45 9mo ago
  So you're saying each line records the new value of the source and destination balance, rather than just the sum that is being exchanged?
  - mrkeen 9mo ago
    No. At the beginning of time, all your accounts will have their starting value. When the first transaction (from,to,value) happens, you will do one overdraft check, and if it's good, you will do 1 addition and 1 subtraction, and two of the accounts will have a new value. On the millionth transaction, you will do one overdraft check, and if it's good, you will do 1 addition and 1 subtraction, and two of the accounts will have a new value. At no point will you need to do more than one check & one add & one sub per arriving transaction. (The append-only is what allows this: the next state is only ever a single, cheap step from the current state. But if someone insists upon mutating history, the current state is no longer valid because it no longer represents the history that led up to it, so it cannot be used to generate the next state - you need to throw it all away and regenerate the current/next states, starting from 0 and replaying every transaction again.