3 ms·

They should have an option to write every block twice including the WAL. SQLite is already so robust, why not go the extra mile Has anyone seen a study of the

They should have an option to write every block twice including the WAL. SQLite is already so robust, why not go the extra mile Has anyone seen a study of the pattern of bit errors experienced on disks? Like are they a single bit or multiple consecutive sectors? At Quantcast in our QFS distributed file system we used a very efficient reed-solomon coding to get redundancy as good as 3x blocks with only 1.5x storage and still got gigabytes/sec encoding speed. We spread blocks out over different cabinets (not just machines or drives) but that would be overkill here. Obviously we still had end to end checksums because errors can happen anywhere, especially in disk and network controllers whose firmware is usually a one off hack

lxgr 2y ago
Why? SQLite has a very well defined and documented set of abstractions about what can and can’t go wrong on the layers of abstraction it relies on. “Out of two writes, at least one will be successful” is not part of that, and doesn’t make sense to me. Second-guessing the storage layer like that will never be able to catch all problems with it (e.g. drives or file systems lying about the durability of write commands).
- paulsutter 2y ago
  Sorry but bit errors happen all the time at scale, and most commonly in controllers. SQLite is used in so many environments for its robustness rather than actually for SQL. Seems odd to argue agsinst it actually Drives, controllers, and file systems lying about write durability is another great reason to add more robustness, good point
  - lxgr 2y ago
    > Drives, controllers, and file systems lying about write durability is another great reason to add more robustness, good point No, the solution for that is to not use these, not to greatly entangle layers via workarounds in the ones above them.
chasil 2y ago
If you really want to do that, both zfs and btrfs can do it at the filesystem layer.
- paulsutter 2y ago
  zfs and btrfs aren’t available in many sqlite deployments

lxgr 2y ago

Why? SQLite has a very well defined and documented set of abstractions about what can and can’t go wrong on the layers of abstraction it relies on. “Out of two writes, at least one will be successful” is not part of that, and doesn’t make sense to me. Second-guessing the storage layer like that will never be able to catch all problems with it (e.g. drives or file systems lying about the durability of write commands).

paulsutter 2y ago
Sorry but bit errors happen all the time at scale, and most commonly in controllers. SQLite is used in so many environments for its robustness rather than actually for SQL. Seems odd to argue agsinst it actually Drives, controllers, and file systems lying about write durability is another great reason to add more robustness, good point
- lxgr 2y ago
  > Drives, controllers, and file systems lying about write durability is another great reason to add more robustness, good point No, the solution for that is to not use these, not to greatly entangle layers via workarounds in the ones above them.

chasil 2y ago

If you really want to do that, both zfs and btrfs can do it at the filesystem layer.

paulsutter 2y ago
zfs and btrfs aren’t available in many sqlite deployments