4 ms·

Thanks! Yes, we have around 6,000+ assertions in TigerBeetle. A few of these were overtight, hence some of the crashes. But those were the assertions doing the

Thanks! Yes, we have around 6,000+ assertions in TigerBeetle. A few of these were overtight, hence some of the crashes. But those were the assertions doing their job, alerting us that we needed to adjust our mental model, which we did. Otherwise, apart from a small correctness bug in an internal testing feature we added (only in our Java client and only for Jepsen to facilitate the audit) there was only one correctness bug found by Jepsen, and it didn’t affect durability. We’ve written about it here: https://tigerbeetle.com/blog/2025-06-06-fuzzer-blind-spots-meet-jepsen/ https://tigerbeetle.com/blog/2025-06-06-fuzzer-blind-spots-m... Finally, to be fair, TigerBeetle can (and is tested) to survive more faults than Postgres can, since it was designed with an explicit storage fault model and using research that was not available at the time when Postgres was released in ‘96. TB’s fault models are further tested with Deterministic Simulation Testing and we use techniques such as static memory allocation following NASA’s Power of Ten Rules for Safety-Critical Code. There are known scenarios in the literature that will cause Postgres to lose data, which TigerBeetle can detect and recover from. For more on this, see the section in Kyle’s report on helical fault injection (most Raft and Paxos implementations were not designed to survive this) as well as a talk we gave at QCon London: https://m.youtube.com/watch?v=_jfOk4L7CiY https://m.youtube.com/watch?v=_jfOk4L7CiY

jrpelkonen 1y ago
Hi Joran, I have followed TigerBeetle with interest for a while, and thank you for your inspirational work and informative presentations. However, you have stated in several occasions that the lack of memory safety in Zig is not a concern since you don't dynamically allocate memory post startup. However, one of the defects uncovered here (#2435) was caused by dereferencing an uninitialized pointer. I find this pretty concerning, so I wonder if there is something that you will be doing differently to eliminate all similar bugs going forward?
- matklad 1y ago
  Note that that's a bug in the client, in the Zig-java FFI code, which is inherently unsafe. We'd likely made an a similar bug in Rust. Which is, yeah, one of the bigger technical challenges for us --- we ship language-native libraries for Go,node,Java,C#,Python and Rust, and, like in the Tolstoi novel, each one is peculiar in its own way. What's worse, they aren't directly covered by our deterministic simulator. That's one of the major reasons why we invest in full-system simulation with jepsen, antithesis and vortex (https://tigerbeetle.com/blog/2025-02-13-a-descent-into-the-vortex/ https://tigerbeetle.com/blog/2025-02-13-a-descent-into-the-v...). We are also toying with the idea of generating _more_ of that code, so there's less room for human error. Maybe one day we'll even do fully native client (eg, pure Java, pure Go), but we are not there yet. One super-specific in-progress thing is that, at the moment, the _bulk_ of the client testing is duplicated per client, and also the _bulk_ of the testing is example-based. Building simulator/workload is a lot of work, and duplicating it for each client is unreasonable. What we want to do here is to use multi-process architecture, where there's a single Zig process that generates the workloads and generates interesting sequences of commands for clients, and than in each client we implement just a tiny "interpreter" for workload language, getting a test suite for free. This is still WIP though! Regarding the broader memory safety issue in the database. We did have a couple of memory safety bugs, which were caught early in testing. We did have one very bad aliasing bug, which would have been totally prevented by Rust, which slipped through the bulk of our testing and into the release (it was caught in testing _after_ it was introduced): https://github.com/tigerbeetle/tigerbeetle/pull/2774 https://github.com/tigerbeetle/tigerbeetle/pull/2774. Notably, while the bug was bad enough to completely mess up our internal data structure, it was immediately caught by an assert down the line, and downgraded from correctness issues to a small availability issues (just restarting the replica would fix it). Curiously, the root cause for that bug was that we over-complicated our code. Long before the actual bug we felt uneasy about the data structure in question, and thought about refactoring it away (which refactor is underway. Hilariously, it looks that just "removing" the thing without any other code changes improves performance!). So, on balance, yeah, Rust would've prevented a small number of easy bugs, and on gnarly bug, but then the entire thing would have to look completely different, as the architecture of TigerBeetle not at all Rust-friendly. I'd be curious to see someone replicating single-thread io-uring no malloc after startup architecture in Rust! I personally don't know off the top of my head whether that would work or not.
  - jcalabro 1y ago
    I remember reading a similar thing about FoundationDB with their DST a while back. Over time, they surfaced relatively few bugs in the core server, but found a bunch in the client libraries because the clients were more complicated and were not run under their DST. Anyways, really interesting report and project. I also like your youtube show - keep up the great work! :)
  - matklad 1y ago
    Oh, important clarification from andrewrk(https://lobste.rs/c/tf6jng https://lobste.rs/c/tf6jng), which I totally missed myself: this isn't actually a dereference of uninitialized pointer, it's a defer of a pointer which is explicitly set to a specific, invalid value.
    - jrpelkonen 1y ago
      This is indeed an important point, the way I originally understood the bug was that the memory was not initialized at all. Thanks for the clarification
    - kiitos 1y ago
      well, per the zig spec, any program that relies on that "explicitly set [and] specific" value of 0xAA isn't valid, so it's absolutely a bug
- AndyKelley 1y ago
  TigerBeetle uses ReleaseSafe optimization mode, which means that the pointer was in fact initialized to 0xaaaaaaaaaaaaaaaa. Since nothing is mapped to this address, it reliably causes a segfault. This is equivalent to an assertion failure.
  - jrpelkonen 1y ago
    That’s good to hear! Thanks for the clarification.
anarazel 1y ago
> There are known scenarios in the literature that will cause Postgres to lose data, which TigerBeetle can detect and recover from. What are you referencing here?
- jorangreef 1y ago
  The scenarios described in our QCon London talk linked above. This surveys the excellent storage fault research from UW-Madison, and in particular: “Can Applications Recover from fsync Failures?” “Protocol-Aware Recovery for Consensus-Based Storage” Finally, I'd recommend watching “Consensus and the Art of Durability”, our talk from SD24 in NYC last year: https://www.youtube.com/watch?v=tRgvaqpQPwE https://www.youtube.com/watch?v=tRgvaqpQPwE
  - kiitos 1y ago
    [disks are] somewhere between non-byzentine fault tolerance and Byzantine fault tolerance ... you expect the disk to be almost an active adversary ... ... so you start to see just a single disk as a distributed system My goodness, not at all! If you can't trust the interface to a local disk then you're lost just at a fundamental level. And even ignoring that, a disk is an implementation detail of a node in a distributed system, whatever properties that disk may have to that local node are irrelevant in the context of the broader system, and are the responsibility of the local node to manage before communicating anything with other nodes in that broader system. Combined with https://www.youtube.com/watch?v=tRgvaqpQPwE https://www.youtube.com/watch?v=tRgvaqpQPwE it seems like the author/presenter is conflating local/disk-related properties/details with distributed/system-based requirements/guarantees. If consensus requires a node to have durably persisted some bit of state before it sends a particular message to other nodes in the distributed system, then it doesn't matter how that persistence is implemented, it only matters how that persistence is observable, disks and FS caches and etc. aren't requirements, they're just one of many possible implementation choices.
    - jorangreef 1y ago
      Recommend you first read the FAST18-winning “Protocol-Aware Recovery for Consensus-Based Storage”. It’s a mindbender of a paradigm-shift for how to think about local recovery actions in the context of the global consensus protocol!
nickpsecurity 1y ago
The correctness bug was due to combinations of features. I'm curious if you've looked into combinatorial testing which NIST claimed knocks out almost all bugs when 6-way testing was used. https://csrc.nist.gov/projects/automated-combinatorial-testing-for-software https://csrc.nist.gov/projects/automated-combinatorial-testi... My intro to other categories of test generation was usually this paper: https://cs.stanford.edu/people/saswat/research/ASTJSS.pdf https://cs.stanford.edu/people/saswat/research/ASTJSS.pdf Maybe see of your team can build combinatorial- or path-based testing in Zig next.
- nickpsecurity 1y ago
  Edit to add 2019, NIST, intro slides on combinatorial testing since it's a better overview: https://csrc.nist.gov/CSRC/media/Projects/Automated-Combinatorial-Testing-for-Software/images-media/dataworks-2019%20.pdf https://csrc.nist.gov/CSRC/media/Projects/Automated-Combinat...
  - jorangreef 1y ago
    Thanks!
schubart 1y ago
> we have around 6,000+ assertions in TigerBeetle. Are they enabled in production? Are there some expensive ones that aren’t?
- jorangreef 1y ago
  Yes, we drive with the seat belts on. It’s not expensive. Because we batch, this naturally separates the control plane from the data plane, amortizing assertions against the (larger) buffers now flowing through the data plane. We do also have some intensive online verification checks, and these are gated behind a comptime flag. Finally, we compile Zig with ReleaseSafe and further have all Zig’s own assertions enabled. For example, checked arithmetic for bounds overflow, which is not something you see enabled by default in safe builds for most languages, but which is critically important for safety. The reason why all this is so important, is because if your program does something wrong in production, with people’s money, you want to know about it immediately and shutdown safely. In other words, production is where you most need the safety, not in development (although you obviously want them there too to find bugs faster). But again, it’s the bugs that make it to production that we’re trying to catch with assertions.
  - schubart 1y ago
    Thanks for your reply! > it’s the bugs that make it to production that we’re trying to catch with assertions. Nicely put, I think I’ll steal this!
    - jorangreef 1y ago
      Great to hear, I'll be using it in future too!

jrpelkonen 1y ago

Hi Joran, I have followed TigerBeetle with interest for a while, and thank you for your inspirational work and informative presentations. However, you have stated in several occasions that the lack of memory safety in Zig is not a concern since you don't dynamically allocate memory post startup. However, one of the defects uncovered here (#2435) was caused by dereferencing an uninitialized pointer. I find this pretty concerning, so I wonder if there is something that you will be doing differently to eliminate all similar bugs going forward?

matklad 1y ago
Note that that's a bug in the client, in the Zig-java FFI code, which is inherently unsafe. We'd likely made an a similar bug in Rust. Which is, yeah, one of the bigger technical challenges for us --- we ship language-native libraries for Go,node,Java,C#,Python and Rust, and, like in the Tolstoi novel, each one is peculiar in its own way. What's worse, they aren't directly covered by our deterministic simulator. That's one of the major reasons why we invest in full-system simulation with jepsen, antithesis and vortex (https://tigerbeetle.com/blog/2025-02-13-a-descent-into-the-vortex/ https://tigerbeetle.com/blog/2025-02-13-a-descent-into-the-v...). We are also toying with the idea of generating _more_ of that code, so there's less room for human error. Maybe one day we'll even do fully native client (eg, pure Java, pure Go), but we are not there yet. One super-specific in-progress thing is that, at the moment, the _bulk_ of the client testing is duplicated per client, and also the _bulk_ of the testing is example-based. Building simulator/workload is a lot of work, and duplicating it for each client is unreasonable. What we want to do here is to use multi-process architecture, where there's a single Zig process that generates the workloads and generates interesting sequences of commands for clients, and than in each client we implement just a tiny "interpreter" for workload language, getting a test suite for free. This is still WIP though! Regarding the broader memory safety issue in the database. We did have a couple of memory safety bugs, which were caught early in testing. We did have one very bad aliasing bug, which would have been totally prevented by Rust, which slipped through the bulk of our testing and into the release (it was caught in testing _after_ it was introduced): https://github.com/tigerbeetle/tigerbeetle/pull/2774 https://github.com/tigerbeetle/tigerbeetle/pull/2774. Notably, while the bug was bad enough to completely mess up our internal data structure, it was immediately caught by an assert down the line, and downgraded from correctness issues to a small availability issues (just restarting the replica would fix it). Curiously, the root cause for that bug was that we over-complicated our code. Long before the actual bug we felt uneasy about the data structure in question, and thought about refactoring it away (which refactor is underway. Hilariously, it looks that just "removing" the thing without any other code changes improves performance!). So, on balance, yeah, Rust would've prevented a small number of easy bugs, and on gnarly bug, but then the entire thing would have to look completely different, as the architecture of TigerBeetle not at all Rust-friendly. I'd be curious to see someone replicating single-thread io-uring no malloc after startup architecture in Rust! I personally don't know off the top of my head whether that would work or not.
- jcalabro 1y ago
  I remember reading a similar thing about FoundationDB with their DST a while back. Over time, they surfaced relatively few bugs in the core server, but found a bunch in the client libraries because the clients were more complicated and were not run under their DST. Anyways, really interesting report and project. I also like your youtube show - keep up the great work! :)
- matklad 1y ago
  Oh, important clarification from andrewrk(https://lobste.rs/c/tf6jng https://lobste.rs/c/tf6jng), which I totally missed myself: this isn't actually a dereference of uninitialized pointer, it's a defer of a pointer which is explicitly set to a specific, invalid value.
  - jrpelkonen 1y ago
    This is indeed an important point, the way I originally understood the bug was that the memory was not initialized at all. Thanks for the clarification
  - kiitos 1y ago
    well, per the zig spec, any program that relies on that "explicitly set [and] specific" value of 0xAA isn't valid, so it's absolutely a bug
AndyKelley 1y ago
TigerBeetle uses ReleaseSafe optimization mode, which means that the pointer was in fact initialized to 0xaaaaaaaaaaaaaaaa. Since nothing is mapped to this address, it reliably causes a segfault. This is equivalent to an assertion failure.
- jrpelkonen 1y ago
  That’s good to hear! Thanks for the clarification.

anarazel 1y ago

> There are known scenarios in the literature that will cause Postgres to lose data, which TigerBeetle can detect and recover from. What are you referencing here?

jorangreef 1y ago
The scenarios described in our QCon London talk linked above. This surveys the excellent storage fault research from UW-Madison, and in particular: “Can Applications Recover from fsync Failures?” “Protocol-Aware Recovery for Consensus-Based Storage” Finally, I'd recommend watching “Consensus and the Art of Durability”, our talk from SD24 in NYC last year: https://www.youtube.com/watch?v=tRgvaqpQPwE https://www.youtube.com/watch?v=tRgvaqpQPwE
- kiitos 1y ago
  [disks are] somewhere between non-byzentine fault tolerance and Byzantine fault tolerance ... you expect the disk to be almost an active adversary ... ... so you start to see just a single disk as a distributed system My goodness, not at all! If you can't trust the interface to a local disk then you're lost just at a fundamental level. And even ignoring that, a disk is an implementation detail of a node in a distributed system, whatever properties that disk may have to that local node are irrelevant in the context of the broader system, and are the responsibility of the local node to manage before communicating anything with other nodes in that broader system. Combined with https://www.youtube.com/watch?v=tRgvaqpQPwE https://www.youtube.com/watch?v=tRgvaqpQPwE it seems like the author/presenter is conflating local/disk-related properties/details with distributed/system-based requirements/guarantees. If consensus requires a node to have durably persisted some bit of state before it sends a particular message to other nodes in the distributed system, then it doesn't matter how that persistence is implemented, it only matters how that persistence is observable, disks and FS caches and etc. aren't requirements, they're just one of many possible implementation choices.
  - jorangreef 1y ago
    Recommend you first read the FAST18-winning “Protocol-Aware Recovery for Consensus-Based Storage”. It’s a mindbender of a paradigm-shift for how to think about local recovery actions in the context of the global consensus protocol!

nickpsecurity 1y ago

The correctness bug was due to combinations of features. I'm curious if you've looked into combinatorial testing which NIST claimed knocks out almost all bugs when 6-way testing was used. https://csrc.nist.gov/projects/automated-combinatorial-testing-for-software https://csrc.nist.gov/projects/automated-combinatorial-testi... My intro to other categories of test generation was usually this paper: https://cs.stanford.edu/people/saswat/research/ASTJSS.pdf https://cs.stanford.edu/people/saswat/research/ASTJSS.pdf Maybe see of your team can build combinatorial- or path-based testing in Zig next.

nickpsecurity 1y ago
Edit to add 2019, NIST, intro slides on combinatorial testing since it's a better overview: https://csrc.nist.gov/CSRC/media/Projects/Automated-Combinatorial-Testing-for-Software/images-media/dataworks-2019%20.pdf https://csrc.nist.gov/CSRC/media/Projects/Automated-Combinat...
- jorangreef 1y ago
  Thanks!

schubart 1y ago

> we have around 6,000+ assertions in TigerBeetle. Are they enabled in production? Are there some expensive ones that aren’t?

jorangreef 1y ago
Yes, we drive with the seat belts on. It’s not expensive. Because we batch, this naturally separates the control plane from the data plane, amortizing assertions against the (larger) buffers now flowing through the data plane. We do also have some intensive online verification checks, and these are gated behind a comptime flag. Finally, we compile Zig with ReleaseSafe and further have all Zig’s own assertions enabled. For example, checked arithmetic for bounds overflow, which is not something you see enabled by default in safe builds for most languages, but which is critically important for safety. The reason why all this is so important, is because if your program does something wrong in production, with people’s money, you want to know about it immediately and shutdown safely. In other words, production is where you most need the safety, not in development (although you obviously want them there too to find bugs faster). But again, it’s the bugs that make it to production that we’re trying to catch with assertions.
- schubart 1y ago
  Thanks for your reply! > it’s the bugs that make it to production that we’re trying to catch with assertions. Nicely put, I think I’ll steal this!
  - jorangreef 1y ago
    Great to hear, I'll be using it in future too!