4 ms·

> There are known scenarios in the literature that will cause Postgres to lose data, which TigerBeetle can detect and recover from. What are you referencing he

> There are known scenarios in the literature that will cause Postgres to lose data, which TigerBeetle can detect and recover from. What are you referencing here?

jorangreef 1y ago
The scenarios described in our QCon London talk linked above. This surveys the excellent storage fault research from UW-Madison, and in particular: “Can Applications Recover from fsync Failures?” “Protocol-Aware Recovery for Consensus-Based Storage” Finally, I'd recommend watching “Consensus and the Art of Durability”, our talk from SD24 in NYC last year: https://www.youtube.com/watch?v=tRgvaqpQPwE https://www.youtube.com/watch?v=tRgvaqpQPwE
- kiitos 1y ago
  [disks are] somewhere between non-byzentine fault tolerance and Byzantine fault tolerance ... you expect the disk to be almost an active adversary ... ... so you start to see just a single disk as a distributed system My goodness, not at all! If you can't trust the interface to a local disk then you're lost just at a fundamental level. And even ignoring that, a disk is an implementation detail of a node in a distributed system, whatever properties that disk may have to that local node are irrelevant in the context of the broader system, and are the responsibility of the local node to manage before communicating anything with other nodes in that broader system. Combined with https://www.youtube.com/watch?v=tRgvaqpQPwE https://www.youtube.com/watch?v=tRgvaqpQPwE it seems like the author/presenter is conflating local/disk-related properties/details with distributed/system-based requirements/guarantees. If consensus requires a node to have durably persisted some bit of state before it sends a particular message to other nodes in the distributed system, then it doesn't matter how that persistence is implemented, it only matters how that persistence is observable, disks and FS caches and etc. aren't requirements, they're just one of many possible implementation choices.
  - jorangreef 1y ago
    Recommend you first read the FAST18-winning “Protocol-Aware Recovery for Consensus-Based Storage”. It’s a mindbender of a paradigm-shift for how to think about local recovery actions in the context of the global consensus protocol!
    - kiitos 1y ago
      I've read that paper for sure! > Disks and flash devices exhibit a subtle and complex failure model: a few blocks of data could become inaccessible or be silently corrupted [8, 9, 32, 59]. Although such storage faults are rare compared to whole-machine failures, in large-scale distributed systems, even rare failures become prevalent [60, 62]. Thus, it is critical to reliably detect and recover from storage faults. It's not true in general that a node in a distributed system binds its persisted state to a local disk, or flash device, or any specific implementation of any specific kind of storage system. The storage layer is the responsibility of the node to manage, and irrelevant to the wider distributed system in which the node is a participant. Any of those kinds of storage faults need to be accommodated by the node that utilizes those storage layers, but their specific details don't need to be communicated beyond the specific node where they apply. And it's not at all critical for those nodes to detect and recover from any faults in their storage layer; those faults can easily be communicated to the broader distributed system, which necessarily must be able to handle node-specific failures like those without breaking everything down! > how to think about local recovery actions in the context of the global consensus protocol! Local recovery actions, or any other kinds of node-specific details, have no relevance or influence on the information communicated thru the the global consensus protocol. A node can persist its state to a local disk, or to RAM, or to an S3 object, or anything else, and none of these details matter at all to the details that node communicates to other nodes in its cluster. tl;dr: nodes don't necessarily persist state to local disk
      - jorangreef 1y ago
        I’m not disagreeing that diskless crash recovery protocols exist. In fact, an early version of TigerBeetle implemented one of these from Cowling and Liskov’s VSR’12 paper. However, since then, we invested in TigerBeetle’s stable storage, for reasons which I won’t go into further here. If you are curious to learn more about all the stable storage techniques in TigerBeetle in particular (again, this is not trying to suggest that VSR can’t also run without stable storage, or to deny other techniques such as object storage or tiering!), but if these things are interesting for you, and if you do want to learn more, then I’d recommend diving into “Durability and the Art of Consensus”.

> There are known scenarios in the literature that will cause Postgres to lose data, which TigerBeetle can detect and recover from. What are you referencing here?

jorangreef 1y ago

The scenarios described in our QCon London talk linked above. This surveys the excellent storage fault research from UW-Madison, and in particular: “Can Applications Recover from fsync Failures?” “Protocol-Aware Recovery for Consensus-Based Storage” Finally, I'd recommend watching “Consensus and the Art of Durability”, our talk from SD24 in NYC last year: https://www.youtube.com/watch?v=tRgvaqpQPwE https://www.youtube.com/watch?v=tRgvaqpQPwE

kiitos 1y ago
[disks are] somewhere between non-byzentine fault tolerance and Byzantine fault tolerance ... you expect the disk to be almost an active adversary ... ... so you start to see just a single disk as a distributed system My goodness, not at all! If you can't trust the interface to a local disk then you're lost just at a fundamental level. And even ignoring that, a disk is an implementation detail of a node in a distributed system, whatever properties that disk may have to that local node are irrelevant in the context of the broader system, and are the responsibility of the local node to manage before communicating anything with other nodes in that broader system. Combined with https://www.youtube.com/watch?v=tRgvaqpQPwE https://www.youtube.com/watch?v=tRgvaqpQPwE it seems like the author/presenter is conflating local/disk-related properties/details with distributed/system-based requirements/guarantees. If consensus requires a node to have durably persisted some bit of state before it sends a particular message to other nodes in the distributed system, then it doesn't matter how that persistence is implemented, it only matters how that persistence is observable, disks and FS caches and etc. aren't requirements, they're just one of many possible implementation choices.
- jorangreef 1y ago
  Recommend you first read the FAST18-winning “Protocol-Aware Recovery for Consensus-Based Storage”. It’s a mindbender of a paradigm-shift for how to think about local recovery actions in the context of the global consensus protocol!
  - kiitos 1y ago
    I've read that paper for sure! > Disks and flash devices exhibit a subtle and complex failure model: a few blocks of data could become inaccessible or be silently corrupted [8, 9, 32, 59]. Although such storage faults are rare compared to whole-machine failures, in large-scale distributed systems, even rare failures become prevalent [60, 62]. Thus, it is critical to reliably detect and recover from storage faults. It's not true in general that a node in a distributed system binds its persisted state to a local disk, or flash device, or any specific implementation of any specific kind of storage system. The storage layer is the responsibility of the node to manage, and irrelevant to the wider distributed system in which the node is a participant. Any of those kinds of storage faults need to be accommodated by the node that utilizes those storage layers, but their specific details don't need to be communicated beyond the specific node where they apply. And it's not at all critical for those nodes to detect and recover from any faults in their storage layer; those faults can easily be communicated to the broader distributed system, which necessarily must be able to handle node-specific failures like those without breaking everything down! > how to think about local recovery actions in the context of the global consensus protocol! Local recovery actions, or any other kinds of node-specific details, have no relevance or influence on the information communicated thru the the global consensus protocol. A node can persist its state to a local disk, or to RAM, or to an S3 object, or anything else, and none of these details matter at all to the details that node communicates to other nodes in its cluster. tl;dr: nodes don't necessarily persist state to local disk
    - jorangreef 1y ago
      I’m not disagreeing that diskless crash recovery protocols exist. In fact, an early version of TigerBeetle implemented one of these from Cowling and Liskov’s VSR’12 paper. However, since then, we invested in TigerBeetle’s stable storage, for reasons which I won’t go into further here. If you are curious to learn more about all the stable storage techniques in TigerBeetle in particular (again, this is not trying to suggest that VSR can’t also run without stable storage, or to deny other techniques such as object storage or tiering!), but if these things are interesting for you, and if you do want to learn more, then I’d recommend diving into “Durability and the Art of Consensus”.