6 ms·

I had the experience of being part of a team in 2020 doing performance and safety analysis on Mojaloop [1], the open-source payments switch. The emphasis was mo

by _vvhw 4y ago

I had the experience of being part of a team in 2020 doing performance and safety analysis on Mojaloop [1], the open-source payments switch. The emphasis was mostly on identifying performance bottlenecks. For example, graphing waterfalls of database queries, estimating expected vs actual concurrency, digging into latency spikes, surprising timeout interactions, and missed group commit opportunities. However, on the safety front, one of the most challenging issues was guaranteeing the rollback of funds in the event of failure as part of the two-phase commit protocol for moving money—coordinating this across multiple SQL queries, database transactions, Kafka queues, even multiple code repositories, especially as different systems experience clock drift or as disks or machines fail. You want to ensure that the money either moves, or doesn't move, that it doesn't get lost somewhere in between. Yet most payment systems re-implement all this business logic on an ad hoc basis, again and again. We therefore extracted these primitives from Mojaloop “once and for all” to create TigerBeetle [2], an open-source financial accounting database, that provides multi-AZ replication, automated leader election, and two-phase payments out-of-the-box. Moving this business logic to the financial database layer opened up three orders of magnitude more performance, with TigerBeetle able to process a million transactions a second, all with an extremely high safety standard. For example, we do FoundationDB-style deterministic simulation testing, but with automated storage fault injection, such as 30% corruption on all replicas including the leader. Design docs, and links to talks are all in the GitHub repo [2]. Happy to answer questions from our experience, or chat if you're working on similar systems! [1] https://mojaloop.io https://mojaloop.io [2] https://github.com/coilhq/tigerbeetle https://github.com/coilhq/tigerbeetle (Zig)

jasfi 4y ago
> Moving this business logic to the financial database layer opened up three orders of magnitude more performance This is because there is an IO cost in sending a request to the DB and then receiving a response. If the DB is on another machine the cost is even higher. Do this enough times and the wait times add up a lot. By shifting business logic to stored procedures you avoid this. That's also why SQLite is very fast, as it runs in your application's memory as a library. But then your data is tied to the same limitations as the machine the application is on.
- _vvhw 4y ago
  > By shifting business logic to stored procedures you avoid this. Thanks, we considered stored procedures to bring the number of database queries down from 18 queries per payment to 1 query per payment. However, that would have provided only an order of magnitude improvement, and brought with it complexity of testing, compared to the state machine [1] we have in TigerBeetle. At the same time, the biggest performance bottleneck is not only the number of roundtrips, but the lack of first-class batching in the interface per roundtrip. What we do in TigerBeetle instead, is we send 8192 transfers in a single network request. This brings the network/disk cost equation down from 1 query per payment, to 1/8192 query per payment. It's like group commit, on steroids. [1] https://github.com/coilhq/tigerbeetle/blob/main/src/state_machine.zig#L331-L406 https://github.com/coilhq/tigerbeetle/blob/main/src/state_ma... > That's also why SQLite is very fast, as it runs in your application's memory as a library. But then your data is tied to the same limitations as the machine the application is on. SQLite is one of my favorite storage engines. However, SQLite does not solve our storage fault model. For example, misdirected reads/writes, lost reads/writes, bitrot in the middle of the committed log. SQLite was also not explicitly designed to be integrated with a global consensus protocol as per ”Protocol-Aware Recovery for Consensus-Based Storage” from UW-Madison. For example, there are optimizations around storage fault tolerance in the commit log that you can do, or around deterministic storage across replicas for faster distributed recovery, that you can't do with SQLite. Check out the paper [2] from UW-Madison for the details, which apply also to LevelDB and RocksDB. We wanted our engine also to be able to run in our deterministic simulator. For example, no random thread scheduling etc. [2] https://www.usenix.org/conference/fast18/presentation/alagappan https://www.usenix.org/conference/fast18/presentation/alagap...
  - jasfi 4y ago
    It sounds like the architecture is different to what I understood. I'll read up more on your system, it sounds very interesting!
    - _vvhw 4y ago
      Thanks! Great to see that you're working on projects in Nim!
zackmorris 4y ago
It sounds like payments might be part of the larger concept of declarative programming (DP): https://stackoverflow.com/questions/129628/what-is-declarative-programming https://stackoverflow.com/questions/129628/what-is-declarati... Maybe TigerBeetle could be generalized to support any multi-step distributed process? I've used DP for backend server work on AWS with Terraform and it gave me perhaps 100x leverage over what I would have been able to do manually. And I think I first heard about it from a friend who used it in Ansible in the 2010s. Now I see everything through that lens and find most of the online tutorials at sites like https://www.raywenderlich.com https://www.raywenderlich.com to be somewhat tedious and perhaps too application-driven (as opposed to theory). Not to single them out - they're one of the best, along with https://laracasts.com https://laracasts.com for backend server concepts. But to really get to scalable solutions, DP is a must IMHO, because it raises attention from implementation details to repeatable processes, which frees the developer to work at a much higher level of abstraction.
- _vvhw 4y ago
  > It sounds like payments might be part of the larger concept of declarative programming (DP) Yes, exactly! The idea with TigerBeetle's state machine [1] is to expose double-entry accounting as a higher level financial primitive, so that developers can think in terms of declaring transfers from one account to another. The business logic behind the scenes is detailed. The interfaces and data structures are simple. [1] https://github.com/coilhq/tigerbeetle/blob/main/src/state_machine.zig#L331-L406 https://github.com/coilhq/tigerbeetle/blob/main/src/state_ma... > Maybe TigerBeetle could be generalized to support any multi-step distributed process? That's part of the plan—that the distributed database framework of TigerBeetle can be used as an ”Iron Man suit” to support any kind of state machine, with high availability and fault tolerance.
  - zackmorris 4y ago
    Oh nice, where has this been my whole life? Keep fighting the good fight!
    - 4y ago
jjoonathan 4y ago
> You want to ensure that the money either moves, or doesn't move, that it doesn't get lost somewhere in between Speaking of which, I finally hit this nightmare scenario -- in one hand I have an international wire proof of payment from a customer, in the other hand I have my bank insisting that they never received a transfer. It's weeks later, I never shipped, and my customer says they never got the money back -- but they haven't been applying pressure to me, so if they are a scammer they are very bad at it. They also know an atypical amount of RF engineering for a scammer. At the moment it really does look to me like the banking system ate his money. I sense a bit of Fear of God motivating your story, something that motivated your management to care -- who is that authority, so that I might get them on my case?
- _vvhw 4y ago
  > I sense a bit of Fear of God motivating your story, something that motivated your management to care -- who is that authority, so that I might get them on my case? Haha! :) The authority is none other than Remzi Arpaci-Dusseau at UW-Madison, along with NASA's Power of 10: Rules for Developing Safety-Critical Code [1]. Storage faults and safety bugs, or the general lack of assertions in so many software projects, do indeed terrify us as a team! It's also all credit to Coil's leadership, for caring and being willing to invest long term in better open source infrastructure for payments for everyone—and particularly Interledger, as an open interoperability protocol, to create an ”open network of networks” for payments, to fix the problem of payments for everyone [2]. I'm glad also to see that sibling comments have given tips on how to unblock that nightmare scenario you encountered! [1] https://en.wikipedia.org/wiki/The_Power_of_10:_Rules_for_Developing_Safety-Critical_Code https://en.wikipedia.org/wiki/The_Power_of_10:_Rules_for_Dev... [2] http://interledger.org/ http://interledger.org/
- enlightens 4y ago
  Are you in the US? https://www.helpwithmybank.gov/index.html https://www.helpwithmybank.gov/index.html That's a website from the Office of the Comptroller of the Currency, the org that regulates US banks. Dig through the topics to find one most applicable, make sure you did everything they recommend, and then hit the contact button. Or just go to the contact page to contact the OCC directly, most of their recommended steps are “call the bank and ask what’s up”. Also good is the CFPB, and while I’m not sure if a business situation will be something they can address, it wouldn't hurt to contact them as well https://www.consumerfinance.gov/ https://www.consumerfinance.gov/
bsaul 4y ago
Since we're on HN and we love talking about PL : how come you chose such a recent language as Zig for something as critical as a paiement system ? I'm sure Zig is, in theory, safer than let's say, C. But doesn't the fact that it's so recent make it more fragile regarding implementation bugs ?
- OmarIsmail 4y ago
  They address it here: https://github.com/coilhq/tigerbeetle/blob/main/docs/DESIGN.md#why-czig https://github.com/coilhq/tigerbeetle/blob/main/docs/DESIGN....
- _vvhw 4y ago
  Thanks for a great question! When we made the decision: 1. We were thinking long term. For example, a distributed database is a significant investment. I love the spirit and orthogonality of C, but we didn't want to pay the safety tax of C over the next 20 years. At the same time, Zig has that same spirit—it's not only a perfect replacement, but a leap forward in developer velocity, power, performance and ergonomics. 2. We realized that TigerBeetle would take 2 years to get to a production release. This meant that our timeline would intersect with Zig's stability—like skating to where the puck is going to be at, or catching the swell as it breaks, rather than riding out the last of an old wave. The number of surfers (quantity) was not a concern. Rather, we were impressed by the sheer quality and early maturity of Zig's compilation story, not to mention the quality of the Zig community in general. For example, it would be hard to find a better std lib crypto than what Zig already has right now, thanks to Frank Denis of libsodium. 3. The design of TigerBeetle is also a single-threaded control plane. We use io_uring for the data plane to eliminate multi-threaded context switches, so multi-threading for async I/O is less of a necessary evil than it used to be. All memory is statically allocated at startup. We never call free() so there are no UAFs. You can start to see why Rust's borrow checker made less sense for our domain than it would have for others. Of course, concurrency bugs can happen on a single thread, but we make use of other techniques to mitigate them. 4. We wanted the open source to be accessible to newcomers wanting to read or contribute to the project. We didn't want to pay the cost of a steep learning curve over the lifetime of the project. Zig turned out to be a hit here, as we've received feedback from engineers who've worked on Spanner or FoundationDB—remarking on Zig's readability, and this has proved to be a force multiplier, as they've come back again and again to the source, and even sent invaluable bug reports. For example, a bug that turned out to be in Apple's O_DSYNC, not even in TigerBeetle. 5. We do exhaustive fuzz testing and deterministic simulation testing from the inside out. See TigerBeetle's VOPR [1] for more details about our internal audit function. 6. We do exhaustive fuzz testing and deterministic simulation testing from the outside in. We're working with Will Wilson's new Antithesis startup https://antithesis.com https://antithesis.com to use their deterministic Linux hypervisor as our external audit function, to test our compiled TigerBeetle binaries in a deterministic cluster environment. This is different to Jepsen and more advanced in so many ways. For example, there's coverage guided fuzzing, and if we find any bugs, we can replay—again and again. 7. Finally, and most importantly, Andrew Kelley's design decisions, and Zig's approach to safety really resonated with us: no macros, no hidden control flow, no unused variables (this has caught several bugs for us already), checked arithmetic enabled by default in safe builds, spatial memory safety, out-of-memory safety, and explicit memory allocators—crucial since TigerBeetle does not do any dynamic memory allocation after startup. Zig is a superbly well-designed language. For example, Zig's comptime is immensely powerful. If you're curious, we also speak to this as a team in the Q&A at the end of the Zig SHOWTIME talk [2] we did last year, and I go into this in detail in a talk I did at the Recurse center last month [3], also touching on why we picked VSR over RAFT or Paxos for the consensus protocol (that's a whole 'nother can of Paxos!). :) P.S. We run a $20k bug bounty for our consensus code. If anyone can find a Zig bug that violates TigerBeetle's consensus or replication, then we have bounties up to $8192. [1] https://github.com/coilhq/tigerbeetle#simulation-tests https://github.com/coilhq/tigerbeetle#simulation-tests [2] https://www.youtube.com/watch?v=BH2jvJ74npM https://www.youtube.com/watch?v=BH2jvJ74npM [3] https://www.youtube.com/watch?v=rNmZZLant9o https://www.youtube.com/watch?v=rNmZZLant9o

by _vvhw 4y ago

jasfi 4y ago

> Moving this business logic to the financial database layer opened up three orders of magnitude more performance This is because there is an IO cost in sending a request to the DB and then receiving a response. If the DB is on another machine the cost is even higher. Do this enough times and the wait times add up a lot. By shifting business logic to stored procedures you avoid this. That's also why SQLite is very fast, as it runs in your application's memory as a library. But then your data is tied to the same limitations as the machine the application is on.

_vvhw 4y ago
> By shifting business logic to stored procedures you avoid this. Thanks, we considered stored procedures to bring the number of database queries down from 18 queries per payment to 1 query per payment. However, that would have provided only an order of magnitude improvement, and brought with it complexity of testing, compared to the state machine [1] we have in TigerBeetle. At the same time, the biggest performance bottleneck is not only the number of roundtrips, but the lack of first-class batching in the interface per roundtrip. What we do in TigerBeetle instead, is we send 8192 transfers in a single network request. This brings the network/disk cost equation down from 1 query per payment, to 1/8192 query per payment. It's like group commit, on steroids. [1] https://github.com/coilhq/tigerbeetle/blob/main/src/state_machine.zig#L331-L406 https://github.com/coilhq/tigerbeetle/blob/main/src/state_ma... > That's also why SQLite is very fast, as it runs in your application's memory as a library. But then your data is tied to the same limitations as the machine the application is on. SQLite is one of my favorite storage engines. However, SQLite does not solve our storage fault model. For example, misdirected reads/writes, lost reads/writes, bitrot in the middle of the committed log. SQLite was also not explicitly designed to be integrated with a global consensus protocol as per ”Protocol-Aware Recovery for Consensus-Based Storage” from UW-Madison. For example, there are optimizations around storage fault tolerance in the commit log that you can do, or around deterministic storage across replicas for faster distributed recovery, that you can't do with SQLite. Check out the paper [2] from UW-Madison for the details, which apply also to LevelDB and RocksDB. We wanted our engine also to be able to run in our deterministic simulator. For example, no random thread scheduling etc. [2] https://www.usenix.org/conference/fast18/presentation/alagappan https://www.usenix.org/conference/fast18/presentation/alagap...
- jasfi 4y ago
  It sounds like the architecture is different to what I understood. I'll read up more on your system, it sounds very interesting!
  - _vvhw 4y ago
    Thanks! Great to see that you're working on projects in Nim!

zackmorris 4y ago

It sounds like payments might be part of the larger concept of declarative programming (DP): https://stackoverflow.com/questions/129628/what-is-declarative-programming https://stackoverflow.com/questions/129628/what-is-declarati... Maybe TigerBeetle could be generalized to support any multi-step distributed process? I've used DP for backend server work on AWS with Terraform and it gave me perhaps 100x leverage over what I would have been able to do manually. And I think I first heard about it from a friend who used it in Ansible in the 2010s. Now I see everything through that lens and find most of the online tutorials at sites like https://www.raywenderlich.com https://www.raywenderlich.com to be somewhat tedious and perhaps too application-driven (as opposed to theory). Not to single them out - they're one of the best, along with https://laracasts.com https://laracasts.com for backend server concepts. But to really get to scalable solutions, DP is a must IMHO, because it raises attention from implementation details to repeatable processes, which frees the developer to work at a much higher level of abstraction.

_vvhw 4y ago
> It sounds like payments might be part of the larger concept of declarative programming (DP) Yes, exactly! The idea with TigerBeetle's state machine [1] is to expose double-entry accounting as a higher level financial primitive, so that developers can think in terms of declaring transfers from one account to another. The business logic behind the scenes is detailed. The interfaces and data structures are simple. [1] https://github.com/coilhq/tigerbeetle/blob/main/src/state_machine.zig#L331-L406 https://github.com/coilhq/tigerbeetle/blob/main/src/state_ma... > Maybe TigerBeetle could be generalized to support any multi-step distributed process? That's part of the plan—that the distributed database framework of TigerBeetle can be used as an ”Iron Man suit” to support any kind of state machine, with high availability and fault tolerance.
- zackmorris 4y ago
  Oh nice, where has this been my whole life? Keep fighting the good fight!
  - 4y ago

jjoonathan 4y ago

> You want to ensure that the money either moves, or doesn't move, that it doesn't get lost somewhere in between Speaking of which, I finally hit this nightmare scenario -- in one hand I have an international wire proof of payment from a customer, in the other hand I have my bank insisting that they never received a transfer. It's weeks later, I never shipped, and my customer says they never got the money back -- but they haven't been applying pressure to me, so if they are a scammer they are very bad at it. They also know an atypical amount of RF engineering for a scammer. At the moment it really does look to me like the banking system ate his money. I sense a bit of Fear of God motivating your story, something that motivated your management to care -- who is that authority, so that I might get them on my case?

_vvhw 4y ago
> I sense a bit of Fear of God motivating your story, something that motivated your management to care -- who is that authority, so that I might get them on my case? Haha! :) The authority is none other than Remzi Arpaci-Dusseau at UW-Madison, along with NASA's Power of 10: Rules for Developing Safety-Critical Code [1]. Storage faults and safety bugs, or the general lack of assertions in so many software projects, do indeed terrify us as a team! It's also all credit to Coil's leadership, for caring and being willing to invest long term in better open source infrastructure for payments for everyone—and particularly Interledger, as an open interoperability protocol, to create an ”open network of networks” for payments, to fix the problem of payments for everyone [2]. I'm glad also to see that sibling comments have given tips on how to unblock that nightmare scenario you encountered! [1] https://en.wikipedia.org/wiki/The_Power_of_10:_Rules_for_Developing_Safety-Critical_Code https://en.wikipedia.org/wiki/The_Power_of_10:_Rules_for_Dev... [2] http://interledger.org/ http://interledger.org/
enlightens 4y ago
Are you in the US? https://www.helpwithmybank.gov/index.html https://www.helpwithmybank.gov/index.html That's a website from the Office of the Comptroller of the Currency, the org that regulates US banks. Dig through the topics to find one most applicable, make sure you did everything they recommend, and then hit the contact button. Or just go to the contact page to contact the OCC directly, most of their recommended steps are “call the bank and ask what’s up”. Also good is the CFPB, and while I’m not sure if a business situation will be something they can address, it wouldn't hurt to contact them as well https://www.consumerfinance.gov/ https://www.consumerfinance.gov/

bsaul 4y ago

Since we're on HN and we love talking about PL : how come you chose such a recent language as Zig for something as critical as a paiement system ? I'm sure Zig is, in theory, safer than let's say, C. But doesn't the fact that it's so recent make it more fragile regarding implementation bugs ?

OmarIsmail 4y ago
They address it here: https://github.com/coilhq/tigerbeetle/blob/main/docs/DESIGN.md#why-czig https://github.com/coilhq/tigerbeetle/blob/main/docs/DESIGN....
_vvhw 4y ago
Thanks for a great question! When we made the decision: 1. We were thinking long term. For example, a distributed database is a significant investment. I love the spirit and orthogonality of C, but we didn't want to pay the safety tax of C over the next 20 years. At the same time, Zig has that same spirit—it's not only a perfect replacement, but a leap forward in developer velocity, power, performance and ergonomics. 2. We realized that TigerBeetle would take 2 years to get to a production release. This meant that our timeline would intersect with Zig's stability—like skating to where the puck is going to be at, or catching the swell as it breaks, rather than riding out the last of an old wave. The number of surfers (quantity) was not a concern. Rather, we were impressed by the sheer quality and early maturity of Zig's compilation story, not to mention the quality of the Zig community in general. For example, it would be hard to find a better std lib crypto than what Zig already has right now, thanks to Frank Denis of libsodium. 3. The design of TigerBeetle is also a single-threaded control plane. We use io_uring for the data plane to eliminate multi-threaded context switches, so multi-threading for async I/O is less of a necessary evil than it used to be. All memory is statically allocated at startup. We never call free() so there are no UAFs. You can start to see why Rust's borrow checker made less sense for our domain than it would have for others. Of course, concurrency bugs can happen on a single thread, but we make use of other techniques to mitigate them. 4. We wanted the open source to be accessible to newcomers wanting to read or contribute to the project. We didn't want to pay the cost of a steep learning curve over the lifetime of the project. Zig turned out to be a hit here, as we've received feedback from engineers who've worked on Spanner or FoundationDB—remarking on Zig's readability, and this has proved to be a force multiplier, as they've come back again and again to the source, and even sent invaluable bug reports. For example, a bug that turned out to be in Apple's O_DSYNC, not even in TigerBeetle. 5. We do exhaustive fuzz testing and deterministic simulation testing from the inside out. See TigerBeetle's VOPR [1] for more details about our internal audit function. 6. We do exhaustive fuzz testing and deterministic simulation testing from the outside in. We're working with Will Wilson's new Antithesis startup https://antithesis.com https://antithesis.com to use their deterministic Linux hypervisor as our external audit function, to test our compiled TigerBeetle binaries in a deterministic cluster environment. This is different to Jepsen and more advanced in so many ways. For example, there's coverage guided fuzzing, and if we find any bugs, we can replay—again and again. 7. Finally, and most importantly, Andrew Kelley's design decisions, and Zig's approach to safety really resonated with us: no macros, no hidden control flow, no unused variables (this has caught several bugs for us already), checked arithmetic enabled by default in safe builds, spatial memory safety, out-of-memory safety, and explicit memory allocators—crucial since TigerBeetle does not do any dynamic memory allocation after startup. Zig is a superbly well-designed language. For example, Zig's comptime is immensely powerful. If you're curious, we also speak to this as a team in the Q&A at the end of the Zig SHOWTIME talk [2] we did last year, and I go into this in detail in a talk I did at the Recurse center last month [3], also touching on why we picked VSR over RAFT or Paxos for the consensus protocol (that's a whole 'nother can of Paxos!). :) P.S. We run a $20k bug bounty for our consensus code. If anyone can find a Zig bug that violates TigerBeetle's consensus or replication, then we have bounties up to $8192. [1] https://github.com/coilhq/tigerbeetle#simulation-tests https://github.com/coilhq/tigerbeetle#simulation-tests [2] https://www.youtube.com/watch?v=BH2jvJ74npM https://www.youtube.com/watch?v=BH2jvJ74npM [3] https://www.youtube.com/watch?v=rNmZZLant9o https://www.youtube.com/watch?v=rNmZZLant9o