3 ms·
Many SAS disks offer larger sector sizes, which in turn allow storing such ECC/checksum data right there. But, what e.g. Btrfs does is to just checksum and raid
by namibj 7y ago
Many SAS disks offer larger sector sizes, which in turn allow storing such ECC/checksum data right there. But, what e.g. Btrfs does is to just checksum and raid, because silent corruption is _really_ rare, and you'd want some amount of raid anyways. Keep in mind that the disk already implements FEC on a low level.
- elcritch 7y agoYes, btrfs works better on embedded wrt to corruption than ext4. ZFS is too hard to setup. But on embedded devices silent corruption isn’t so rare. After 2-3 months of usage on an SD card it’s almost guaranteed in my experience from last year. USB thumb drives & eMMC fair much better, but I’ve still lost entire DB’s. Given all of that, when corruption does happen you can frustratingly (rocksdb at least) loose an entire dB due to a single checksum error. It’s good rocksdb has checksums but an FEC code could both checksum, and be used to correct the data. Unfortunately off-device backups aren’t an option.
- zaarn 7y agoMight be an idea to offload that. You could reduce the risk of corruption by splitting the database over timeframes, so every 3 weeks, you create a new database, copy all currently relevant data and close out the old one. In case of corruption you can recover from an older version of the database, which is less likely to be corrupted the more versions you keep and the more often you do this (at the expense of flash lifetime).
- namibj 7y agoIf you have the space for storing data twice, you can actually use `dup` for the data, not just the metadata. That conversion can be done online, though I managed to make it zero out all superblocks on all 3 disks while converting from raid1 to raid5. That's some time back, however, and there was some unfortunate power cycling.