4 ms·

Wow. Just starting to watch the talk. Very nice, dense introduction with all the questions I have asked myself about sharding. Let's see if I get how you approa

by weitzj 5y ago

Wow. Just starting to watch the talk. Very nice, dense introduction with all the questions I have asked myself about sharding. Let's see if I get how you approach LMAX, multiple regions, (and what not)

jorangreef 5y ago
Awesome. I'll be around here if I can fill in any gaps we missed for you. Another difficulty with sharding payments is that a transfer can be between a tuple of any two parties. Adrian is the expert in the payments space and can go into this better than I can. TigerBeetle is still alpha and we're moving to beta now with automated fault injection testing of our consensus and replication. The project has been a year since ProtoBeetle in July 2020. Regarding multiple regions, we have some novel Viewstamped Replication optimizations that we hope to write up soon and document in our repo.
- weitzj 5y ago
  Still digesting the talk. Did you have a look at cockroachdb (or their Blog) when thinking of your implementation?
  - jorangreef 5y ago
    We looked to LMAX for the design, and to UW-Madison for our fault models, but appreciated how CockroachDB approached and cared about clock faults. However, our motivation for managing clock faults was not because we wanted to do leader leases. We don't risk stale reads so we don't do leader leases, i.e. where the leader can operate as a lone ranger without consensus from the cluster for a period of time. We're not comfortable with what could potentially go wrong when stale reads are allowed to become part of the fault model. The motivation for having some kind of clock fault tolerance was rather so that TigerBeetle can clamp the system time to cluster time for reasonably accurate transaction timestamps for auditing purposes, and for reliable timeouts of our two-phase commit ledger transfers so that liquidity is not locked up forever. In other words, the only reason we synchronize clocks across the cluster is so that a leader with a far-future clock can't push the cluster's monotonic clock out into the far future for unrealistic timestamps or to delay transaction timeouts (or expire them too early if a new leader's clock is stuck in the past).
- weitzj 5y ago
  Do you have plans to have policies on accounts? I.e. you don’t know your Customer yet fully but allow some small transactions as long as your customer’s account maybe dies not exceed a certain balance. Or is this kind of contract not feasible since you would have to have an order guarantee. Or when will transactions fail logically? Like at a best effort you schedule transactions and if they don’t meet your policies they will marked as a failed transaction. I guess I don’t grasp yet if one needs a gigantic ledger and a replicated state machine and have every transaction applied sequentially (across all machines), or there is some other way
  - jorangreef 5y ago
    Yes, we've reserved space in our Account data type for future accounting policy: https://github.com/coilhq/tigerbeetle/blob/main/src/tigerbeetle.zig#L6-L20 https://github.com/coilhq/tigerbeetle/blob/main/src/tigerbee... At present, TigerBeetle can already be used to limit debits/credits against an account to not exceed the credits/debits of the account by setting account.debits_must_not_exceed_credits = true when creating the account. This will then become an immutable policy on the account and TigerBeetle will then cap transactions to never exceed the liquidity available. If you wanted to rate limit transactions, you could create something like a "prepaid" account where you periodically top up the account, and then let transfers drain it out. Would these address something like what you were thinking? TigerBeetle also has support for something called Linked Events where you can submit a batch of either accounts, transfers or commits and where you can then chain a subset of events within this batch together by toggling a "linked" flag on each, so that these subset chains succeed or fail atomically together, much like what you can do when submitting IO through io_uring: https://github.com/coilhq/tigerbeetle/blob/main/src/tigerbeetle.zig#L56-L66 https://github.com/coilhq/tigerbeetle/blob/main/src/tigerbee... We didn't go into our accounting features too much beyond the big idea because we wanted to focus more on the technical aspects for this talk.
    - weitzj 5y ago
      I was thinking about KYC and my idea was something like: You want to promote your micropayments and therefore give your customers for example $1 as a gift and they just need an email address to sign up on your platform. Once they have $10 in their account they have to at least give you their phone number/address before they can trade more money. Now you have 12 customers sending money to customer A who has now $12 in their account. In the meantime customer A sent out $1 to somebody else. Depending on who sent, got the money first you have breached the $10 limit. And I don’t grasp how this use case can be met with double bookkeeping besides using a single synchronous ledger (or maybe virtual accounts to do reconciliation). If you now have customer A maybe in USA and customer B on another continent I am even more clueless about latency. :)

by weitzj 5y ago

jorangreef 5y ago

Awesome. I'll be around here if I can fill in any gaps we missed for you. Another difficulty with sharding payments is that a transfer can be between a tuple of any two parties. Adrian is the expert in the payments space and can go into this better than I can. TigerBeetle is still alpha and we're moving to beta now with automated fault injection testing of our consensus and replication. The project has been a year since ProtoBeetle in July 2020. Regarding multiple regions, we have some novel Viewstamped Replication optimizations that we hope to write up soon and document in our repo.

weitzj 5y ago
Still digesting the talk. Did you have a look at cockroachdb (or their Blog) when thinking of your implementation?
- jorangreef 5y ago
  We looked to LMAX for the design, and to UW-Madison for our fault models, but appreciated how CockroachDB approached and cared about clock faults. However, our motivation for managing clock faults was not because we wanted to do leader leases. We don't risk stale reads so we don't do leader leases, i.e. where the leader can operate as a lone ranger without consensus from the cluster for a period of time. We're not comfortable with what could potentially go wrong when stale reads are allowed to become part of the fault model. The motivation for having some kind of clock fault tolerance was rather so that TigerBeetle can clamp the system time to cluster time for reasonably accurate transaction timestamps for auditing purposes, and for reliable timeouts of our two-phase commit ledger transfers so that liquidity is not locked up forever. In other words, the only reason we synchronize clocks across the cluster is so that a leader with a far-future clock can't push the cluster's monotonic clock out into the far future for unrealistic timestamps or to delay transaction timeouts (or expire them too early if a new leader's clock is stuck in the past).
weitzj 5y ago
Do you have plans to have policies on accounts? I.e. you don’t know your Customer yet fully but allow some small transactions as long as your customer’s account maybe dies not exceed a certain balance. Or is this kind of contract not feasible since you would have to have an order guarantee. Or when will transactions fail logically? Like at a best effort you schedule transactions and if they don’t meet your policies they will marked as a failed transaction. I guess I don’t grasp yet if one needs a gigantic ledger and a replicated state machine and have every transaction applied sequentially (across all machines), or there is some other way
- jorangreef 5y ago
  Yes, we've reserved space in our Account data type for future accounting policy: https://github.com/coilhq/tigerbeetle/blob/main/src/tigerbeetle.zig#L6-L20 https://github.com/coilhq/tigerbeetle/blob/main/src/tigerbee... At present, TigerBeetle can already be used to limit debits/credits against an account to not exceed the credits/debits of the account by setting account.debits_must_not_exceed_credits = true when creating the account. This will then become an immutable policy on the account and TigerBeetle will then cap transactions to never exceed the liquidity available. If you wanted to rate limit transactions, you could create something like a "prepaid" account where you periodically top up the account, and then let transfers drain it out. Would these address something like what you were thinking? TigerBeetle also has support for something called Linked Events where you can submit a batch of either accounts, transfers or commits and where you can then chain a subset of events within this batch together by toggling a "linked" flag on each, so that these subset chains succeed or fail atomically together, much like what you can do when submitting IO through io_uring: https://github.com/coilhq/tigerbeetle/blob/main/src/tigerbeetle.zig#L56-L66 https://github.com/coilhq/tigerbeetle/blob/main/src/tigerbee... We didn't go into our accounting features too much beyond the big idea because we wanted to focus more on the technical aspects for this talk.
  - weitzj 5y ago
    I was thinking about KYC and my idea was something like: You want to promote your micropayments and therefore give your customers for example $1 as a gift and they just need an email address to sign up on your platform. Once they have $10 in their account they have to at least give you their phone number/address before they can trade more money. Now you have 12 customers sending money to customer A who has now $12 in their account. In the meantime customer A sent out $1 to somebody else. Depending on who sent, got the money first you have breached the $10 limit. And I don’t grasp how this use case can be met with double bookkeeping besides using a single synchronous ledger (or maybe virtual accounts to do reconciliation). If you now have customer A maybe in USA and customer B on another continent I am even more clueless about latency. :)