5 ms·

I was thinking about KYC and my idea was something like: You want to promote your micropayments and therefore give your customers for example $1 as a gift and

by weitzj 5y ago

I was thinking about KYC and my idea was something like: You want to promote your micropayments and therefore give your customers for example $1 as a gift and they just need an email address to sign up on your platform. Once they have $10 in their account they have to at least give you their phone number/address before they can trade more money. Now you have 12 customers sending money to customer A who has now $12 in their account. In the meantime customer A sent out $1 to somebody else. Depending on who sent, got the money first you have breached the $10 limit. And I don’t grasp how this use case can be met with double bookkeeping besides using a single synchronous ledger (or maybe virtual accounts to do reconciliation). If you now have customer A maybe in USA and customer B on another continent I am even more clueless about latency. :)

jorangreef 5y ago
Thanks for the interesting example. If the goal of the policy was to prevent customer A from staying under the KYC threshold by constantly shifting received gifts elsewhere, then I think it should be possible to account for this by giving customer A two accounts, one that they receive gifts into (capped to be no more than incoming 10 credits), and one that they send from (capped to be no more than 1 outgoing debit). KYC would then be required before customer A can receive any more gifts or be transferred into a full-duplex account. Can you spot any bugs in this approach? There might also be better ways to model a chart of accounts for this scenario. And if you have any more scenarios you can imagine and send through, I would also really like to work through them to see what we can learn. Regarding latency: If payments are global, then I believe that any system would need to incur latency at some point when updating both accounts if they are homed in different jurisdictions. At this point, is your question about global cross-continent latency differences between active Paxos replication vs passive Multi-Paxos replication? Or are you more comparing the latency of a strict serializability RSM like TigerBeetle vs the latency of something like a byzantine fault tolerant payment system that uses cryptographic signing and sharding but which cannot do things like payment-versus-payment?
- weitzj 5y ago
  I was more interested in latency of a strict serializability RSM like TigerBeetle. I was thinking that maybe my use case is maybe something which should not be solved in the ledger directly, but merely the ledger should offer an API to fulfill transactions in the ledger. And the Api consumer has to provide an idempotency key to perform the transaction . But then again I would assume that the ledger would have to provide an endpoint what the current balance is (the sum over all transactions). And when the api consumer makes a decision depending on KYC and current balance, whether to try to record a transaction on the ledger, in the meantime the balance could have changed. So maybe one could send an idempotency key and the balance which was used to decide whether to fulfil the transaction. Or a more generic version: an api consumer could send an idempotency key and a smart contract compute kernel on a transaction request. And the smart contract gets evaluated on the fly with the live data inside the ledger. I smell like I am inventing a database and SQL here. ;) Regarding: “ KYC would then be required before customer A can receive any more gifts ” Where would the validation live? Inside the ledger as a contract per customer across their accounts? When customer B tries to send money to customer A and A fails: what will customer B see? An immediate error or an asynchronous message
  - jorangreef 5y ago
    "I was more interested in latency of a strict serializability RSM like TigerBeetle." The back-of-the-envelope calculation for the latency per 10,000 transfers would be: RTT from leader to the fastest follower + ms to send 1.2 MiB over a 1 Gbps or 10 Gbps network + ms to fsync 1.2 MiB on NVME And multiple transfer batches are pipelined or overlapped by the leader to the followers (we're working on pipelining at the moment) to amortize network transfer. The cross-region replication can be asynchronous (configurable) which is also how I believe LMAX do it, or you could have your backup region be a few hundred miles away as with AWS regions. "But then again I would assume that the ledger would have to provide an endpoint what the current balance is (the sum over all transactions)." Exactly. One of TigerBeetle's contributions is to colocate data and policy across a chart of accounts. It's different from a "ledger" in that TigerBeetle is not only accounting books but also accounting policy. The system of record is the optimal and safest place to make decisions, and this avoids the overhead of distributed transactions. This provides a 50% throughput improvement just by cutting out the unnecessary interim query. Instead of amplifying network queries to make decisions away from the data, you design a chart of accounts with simpler primitives to control transfer flows between them. For long-lived transactions that span multiple external systems, TigerBeetle provides two-phase transfers as atomic primitive, which is what many payment systems use, as I am sure you would know. These two-phase transfers also support 256-bit SHA256 hash locks, e.g. ILPv4 conditions, which I think could be used to support some interesting logic outside the database. This all enables the application layer to be completely stateless and scaled shared nothing. You could literally spin up a hundred API instances in front of TigerBeetle, all shared nothing and ephemeral. "When customer B tries to send money to customer A and A fails: what will customer B see?" An immediate error: "exceeds_debits". This can trigger the KYC user flow to be queued in an external message broker service. You can see exactly how this all works here: https://github.com/coilhq/tigerbeetle/blob/main/src/state_machine.zig#L291-L355 https://github.com/coilhq/tigerbeetle/blob/main/src/state_ma...

by weitzj 5y ago

jorangreef 5y ago

Thanks for the interesting example. If the goal of the policy was to prevent customer A from staying under the KYC threshold by constantly shifting received gifts elsewhere, then I think it should be possible to account for this by giving customer A two accounts, one that they receive gifts into (capped to be no more than incoming 10 credits), and one that they send from (capped to be no more than 1 outgoing debit). KYC would then be required before customer A can receive any more gifts or be transferred into a full-duplex account. Can you spot any bugs in this approach? There might also be better ways to model a chart of accounts for this scenario. And if you have any more scenarios you can imagine and send through, I would also really like to work through them to see what we can learn. Regarding latency: If payments are global, then I believe that any system would need to incur latency at some point when updating both accounts if they are homed in different jurisdictions. At this point, is your question about global cross-continent latency differences between active Paxos replication vs passive Multi-Paxos replication? Or are you more comparing the latency of a strict serializability RSM like TigerBeetle vs the latency of something like a byzantine fault tolerant payment system that uses cryptographic signing and sharding but which cannot do things like payment-versus-payment?

weitzj 5y ago
I was more interested in latency of a strict serializability RSM like TigerBeetle. I was thinking that maybe my use case is maybe something which should not be solved in the ledger directly, but merely the ledger should offer an API to fulfill transactions in the ledger. And the Api consumer has to provide an idempotency key to perform the transaction . But then again I would assume that the ledger would have to provide an endpoint what the current balance is (the sum over all transactions). And when the api consumer makes a decision depending on KYC and current balance, whether to try to record a transaction on the ledger, in the meantime the balance could have changed. So maybe one could send an idempotency key and the balance which was used to decide whether to fulfil the transaction. Or a more generic version: an api consumer could send an idempotency key and a smart contract compute kernel on a transaction request. And the smart contract gets evaluated on the fly with the live data inside the ledger. I smell like I am inventing a database and SQL here. ;) Regarding: “ KYC would then be required before customer A can receive any more gifts ” Where would the validation live? Inside the ledger as a contract per customer across their accounts? When customer B tries to send money to customer A and A fails: what will customer B see? An immediate error or an asynchronous message
- jorangreef 5y ago
  "I was more interested in latency of a strict serializability RSM like TigerBeetle." The back-of-the-envelope calculation for the latency per 10,000 transfers would be: RTT from leader to the fastest follower + ms to send 1.2 MiB over a 1 Gbps or 10 Gbps network + ms to fsync 1.2 MiB on NVME And multiple transfer batches are pipelined or overlapped by the leader to the followers (we're working on pipelining at the moment) to amortize network transfer. The cross-region replication can be asynchronous (configurable) which is also how I believe LMAX do it, or you could have your backup region be a few hundred miles away as with AWS regions. "But then again I would assume that the ledger would have to provide an endpoint what the current balance is (the sum over all transactions)." Exactly. One of TigerBeetle's contributions is to colocate data and policy across a chart of accounts. It's different from a "ledger" in that TigerBeetle is not only accounting books but also accounting policy. The system of record is the optimal and safest place to make decisions, and this avoids the overhead of distributed transactions. This provides a 50% throughput improvement just by cutting out the unnecessary interim query. Instead of amplifying network queries to make decisions away from the data, you design a chart of accounts with simpler primitives to control transfer flows between them. For long-lived transactions that span multiple external systems, TigerBeetle provides two-phase transfers as atomic primitive, which is what many payment systems use, as I am sure you would know. These two-phase transfers also support 256-bit SHA256 hash locks, e.g. ILPv4 conditions, which I think could be used to support some interesting logic outside the database. This all enables the application layer to be completely stateless and scaled shared nothing. You could literally spin up a hundred API instances in front of TigerBeetle, all shared nothing and ephemeral. "When customer B tries to send money to customer A and A fails: what will customer B see?" An immediate error: "exceeds_debits". This can trigger the KYC user flow to be queued in an external message broker service. You can see exactly how this all works here: https://github.com/coilhq/tigerbeetle/blob/main/src/state_machine.zig#L291-L355 https://github.com/coilhq/tigerbeetle/blob/main/src/state_ma...