5 ms·

TigerBeetle – A Million Financial Transactions per Second in Zig

41 points by lonjil 5y ago

the_laka 5y ago
That's impressive! A million tps.
- jorangreef 5y ago
  Thanks! It's on the low end of what systems in other domains like LMAX have done, but can make a difference for the payment systems that we worked on in coming up with the design, to go faster and reduce costs. Ahead of performance, we are especially excited about TigerBeetle's safety fault models, where we think there's an opportunity to break new ground, with all the new storage fault research that's been coming out the past two or three years, especially from UW-Madison. Zig has been fantastic, in particular for Direct I/O alignment and static allocation, and it's only getting better with things like the self-hosted compiler. Having done a bit of security work in the past, I really like Zig's unique approach to safety, for example checked arithmetic enabled by default in safe builds, all the explicitness around return values and exhaustive syscall error handling, and of course comptime. The speed and ease of the compiler itself and cross-compilation tooling is excellent and the readability of Zig is also remarkable. Huge props to the Zig Software Foundation for what they are achieving as a fully open source community funded project.
deleted 5y ago
[deleted]
jorangreef 5y ago
Hey everyone, I'm on what we like to call "the TigerTeam" and happy to bounce ideas. Thanks also to Loris Cro for hosting us on Zig ShowTime. We had a blast! :)
weitzj 5y ago
Wow. Just starting to watch the talk. Very nice, dense introduction with all the questions I have asked myself about sharding. Let's see if I get how you approach LMAX, multiple regions, (and what not)
- jorangreef 5y ago
  Awesome. I'll be around here if I can fill in any gaps we missed for you. Another difficulty with sharding payments is that a transfer can be between a tuple of any two parties. Adrian is the expert in the payments space and can go into this better than I can. TigerBeetle is still alpha and we're moving to beta now with automated fault injection testing of our consensus and replication. The project has been a year since ProtoBeetle in July 2020. Regarding multiple regions, we have some novel Viewstamped Replication optimizations that we hope to write up soon and document in our repo.
  - weitzj 5y ago
    Still digesting the talk. Did you have a look at cockroachdb (or their Blog) when thinking of your implementation?
    - jorangreef 5y ago
      We looked to LMAX for the design, and to UW-Madison for our fault models, but appreciated how CockroachDB approached and cared about clock faults. However, our motivation for managing clock faults was not because we wanted to do leader leases. We don't risk stale reads so we don't do leader leases, i.e. where the leader can operate as a lone ranger without consensus from the cluster for a period of time. We're not comfortable with what could potentially go wrong when stale reads are allowed to become part of the fault model. The motivation for having some kind of clock fault tolerance was rather so that TigerBeetle can clamp the system time to cluster time for reasonably accurate transaction timestamps for auditing purposes, and for reliable timeouts of our two-phase commit ledger transfers so that liquidity is not locked up forever. In other words, the only reason we synchronize clocks across the cluster is so that a leader with a far-future clock can't push the cluster's monotonic clock out into the far future for unrealistic timestamps or to delay transaction timeouts (or expire them too early if a new leader's clock is stuck in the past).
  - weitzj 5y ago
    Do you have plans to have policies on accounts? I.e. you don’t know your Customer yet fully but allow some small transactions as long as your customer’s account maybe dies not exceed a certain balance. Or is this kind of contract not feasible since you would have to have an order guarantee. Or when will transactions fail logically? Like at a best effort you schedule transactions and if they don’t meet your policies they will marked as a failed transaction. I guess I don’t grasp yet if one needs a gigantic ledger and a replicated state machine and have every transaction applied sequentially (across all machines), or there is some other way
fithisux 5y ago
I hope we see similar numbers in Dlang or Nim.
potamic 5y ago
A million tps is crazy! I have heard of trading platforms operating at this scale. Curious about your use cases that led to the creation of this.
- jorangreef 5y ago
  Thanks! We had a link to a document in the deck (in a very small font!) that goes into more detail on the work that led to TigerBeetle [1]. More performance allows you to trade expensive hardware for cheaper commodity hardware to gain lower operational cost (e.g. a cluster of three Raspberry Pi's) while still having more than enough headroom to handle volume spikes safely (e.g. Black Friday). With digital payments, payments are also becoming more global, with higher volume and lower value, but existing infrastructure is not typically designed for this high volume low value performance profile (e.g. millions of payments at a fraction of a cent). Finally, high throughput allows you to easily ingest huge amounts of historical data into TigerBeetle in a few seconds. The design actually made this kind of throughput fairly easy to achieve, even for three small office laptops. The hard part is all the safety work that we're doing to make this hopefully many times safer than existing systems. [1] "Coil’s Mojaloop Performance Work 2020" - https://docs.mojaloop.io/documentation/discussions/Mojaloop%20Performance%202020.pdf https://docs.mojaloop.io/documentation/discussions/Mojaloop%...

41 points by lonjil 5y ago

the_laka 5y ago

That's impressive! A million tps.

jorangreef 5y ago
Thanks! It's on the low end of what systems in other domains like LMAX have done, but can make a difference for the payment systems that we worked on in coming up with the design, to go faster and reduce costs. Ahead of performance, we are especially excited about TigerBeetle's safety fault models, where we think there's an opportunity to break new ground, with all the new storage fault research that's been coming out the past two or three years, especially from UW-Madison. Zig has been fantastic, in particular for Direct I/O alignment and static allocation, and it's only getting better with things like the self-hosted compiler. Having done a bit of security work in the past, I really like Zig's unique approach to safety, for example checked arithmetic enabled by default in safe builds, all the explicitness around return values and exhaustive syscall error handling, and of course comptime. The speed and ease of the compiler itself and cross-compilation tooling is excellent and the readability of Zig is also remarkable. Huge props to the Zig Software Foundation for what they are achieving as a fully open source community funded project.

deleted 5y ago

[deleted]

jorangreef 5y ago

Hey everyone, I'm on what we like to call "the TigerTeam" and happy to bounce ideas. Thanks also to Loris Cro for hosting us on Zig ShowTime. We had a blast! :)

weitzj 5y ago

Wow. Just starting to watch the talk. Very nice, dense introduction with all the questions I have asked myself about sharding. Let's see if I get how you approach LMAX, multiple regions, (and what not)

jorangreef 5y ago
Awesome. I'll be around here if I can fill in any gaps we missed for you. Another difficulty with sharding payments is that a transfer can be between a tuple of any two parties. Adrian is the expert in the payments space and can go into this better than I can. TigerBeetle is still alpha and we're moving to beta now with automated fault injection testing of our consensus and replication. The project has been a year since ProtoBeetle in July 2020. Regarding multiple regions, we have some novel Viewstamped Replication optimizations that we hope to write up soon and document in our repo.
- weitzj 5y ago
  Still digesting the talk. Did you have a look at cockroachdb (or their Blog) when thinking of your implementation?
  - jorangreef 5y ago
    We looked to LMAX for the design, and to UW-Madison for our fault models, but appreciated how CockroachDB approached and cared about clock faults. However, our motivation for managing clock faults was not because we wanted to do leader leases. We don't risk stale reads so we don't do leader leases, i.e. where the leader can operate as a lone ranger without consensus from the cluster for a period of time. We're not comfortable with what could potentially go wrong when stale reads are allowed to become part of the fault model. The motivation for having some kind of clock fault tolerance was rather so that TigerBeetle can clamp the system time to cluster time for reasonably accurate transaction timestamps for auditing purposes, and for reliable timeouts of our two-phase commit ledger transfers so that liquidity is not locked up forever. In other words, the only reason we synchronize clocks across the cluster is so that a leader with a far-future clock can't push the cluster's monotonic clock out into the far future for unrealistic timestamps or to delay transaction timeouts (or expire them too early if a new leader's clock is stuck in the past).
- weitzj 5y ago
  Do you have plans to have policies on accounts? I.e. you don’t know your Customer yet fully but allow some small transactions as long as your customer’s account maybe dies not exceed a certain balance. Or is this kind of contract not feasible since you would have to have an order guarantee. Or when will transactions fail logically? Like at a best effort you schedule transactions and if they don’t meet your policies they will marked as a failed transaction. I guess I don’t grasp yet if one needs a gigantic ledger and a replicated state machine and have every transaction applied sequentially (across all machines), or there is some other way

fithisux 5y ago

I hope we see similar numbers in Dlang or Nim.

potamic 5y ago

A million tps is crazy! I have heard of trading platforms operating at this scale. Curious about your use cases that led to the creation of this.

jorangreef 5y ago
Thanks! We had a link to a document in the deck (in a very small font!) that goes into more detail on the work that led to TigerBeetle [1]. More performance allows you to trade expensive hardware for cheaper commodity hardware to gain lower operational cost (e.g. a cluster of three Raspberry Pi's) while still having more than enough headroom to handle volume spikes safely (e.g. Black Friday). With digital payments, payments are also becoming more global, with higher volume and lower value, but existing infrastructure is not typically designed for this high volume low value performance profile (e.g. millions of payments at a fraction of a cent). Finally, high throughput allows you to easily ingest huge amounts of historical data into TigerBeetle in a few seconds. The design actually made this kind of throughput fairly easy to achieve, even for three small office laptops. The hard part is all the safety work that we're doing to make this hopefully many times safer than existing systems. [1] "Coil’s Mojaloop Performance Work 2020" - https://docs.mojaloop.io/documentation/discussions/Mojaloop%20Performance%202020.pdf https://docs.mojaloop.io/documentation/discussions/Mojaloop%...