35 ms·

Disks can buffer, disks can have firmware bugs, disks can fail both catastrophically and subtly. Even the fanciest battery-backed RAID controllers have firmware

Disks can buffer, disks can have firmware bugs, disks can fail both catastrophically and subtly. Even the fanciest battery-backed RAID controllers have firmware bugs and dead batteries. Those are just ways your storage hardware can fail you even if the kernel and libraries are bug free and you follow the right mystic incantations to sync data. While there's no excuse for bad documentation or poor APIs, you can never consider data written to a single local disk "safe". It never is. It's a shame making a best effort at safety is nontrivial, but it does force developers to write more defensive and crash-safe code which is all that can save you in the end.

GauntletWizard 11y ago
Yeah, from a SRE perspective, the last N writes are always purely probabilistic. The real quest is to have enough redundancy that that curve is fairly close to 1, and enough failure warning that your system will fix itself before it droops. That means Battery Backup, SMART detection, ECC memory, etc.
- planckscnst 11y ago
  It's even worse than that. All writes have some probability of being incorrect, even after having been written to disk properly (the programmer did the right thing, the OS did the right thing, the filesystem did the right thing, and the disk did the right thing). After writing, your data can be modified by actions on other cells or by simply leaving it alone completely (see read disturbance and charge leakage). Some of this can be fixed by error correcting codes, but there is still a chance of loosing data simply by doing nothing wrong. So yes, agreed++. You need to have a level of redundancy appropriate to the criticality of the data.
memracom 11y ago
EBS, Riak, etc. Lots of easy solutions to this problem of putting the data in several diverse places with one write call. Use what works and don't complain that broken stuff is broken.

GauntletWizard 11y ago

Yeah, from a SRE perspective, the last N writes are always purely probabilistic. The real quest is to have enough redundancy that that curve is fairly close to 1, and enough failure warning that your system will fix itself before it droops. That means Battery Backup, SMART detection, ECC memory, etc.

planckscnst 11y ago
It's even worse than that. All writes have some probability of being incorrect, even after having been written to disk properly (the programmer did the right thing, the OS did the right thing, the filesystem did the right thing, and the disk did the right thing). After writing, your data can be modified by actions on other cells or by simply leaving it alone completely (see read disturbance and charge leakage). Some of this can be fixed by error correcting codes, but there is still a chance of loosing data simply by doing nothing wrong. So yes, agreed++. You need to have a level of redundancy appropriate to the criticality of the data.

memracom 11y ago

EBS, Riak, etc. Lots of easy solutions to this problem of putting the data in several diverse places with one write call. Use what works and don't complain that broken stuff is broken.