It's possible thanks to “Protocol-Aware Recovery for Consensus-Based Storage” ('18), which TigerBeetle uses to leverage the global redundancy available in the c

It's possible thanks to “Protocol-Aware Recovery for Consensus-Based Storage” ('18), which TigerBeetle uses to leverage the global redundancy available in the consensus protocol to recover from storage faults in the local storage engine (it's at least 3x more cost-efficient to do this, and stronger compared to logical RAID, if you already have replicated durability that you can tap into like we do). UW-Madison have done some terrific research on disk firmware (and filesystem) bugs. As a recent example, there was also a bug in XFS in May that could result in misdirected writes IIRC: https://bugzilla.redhat.com/show_bug.cgi?id=2208553 https://bugzilla.redhat.com/show_bug.cgi?id=2208553. With the storage fault model clearly defined, TigerBeetle's storage fault model can then be tested, with storage fault injection on the read/write path, but at much higher fault probabilities, to see how storage faults interact with the storage engine and consensus protocol. For example, 8-9% chance per I/O on the read and write path, and with the simulator aware of "f", i.e. how many storage faults the simulator can inject across replicas while expecting (and asserting) the consensus protocol to remain available. We normally run our simulator on the command line, but as a fun hack, we used Zig to compile TigerBeetle to WASM and then drew graphics to hook into the real events, so you can see a whole simulated virtual cluster running purely client side in a browser tab: https://tigerbeetle.com/blog/2023-07-11-we-put-a-distributed-database-in-the-browser/ https://tigerbeetle.com/blog/2023-07-11-we-put-a-distributed...