2 ms·
A logical issue that I have with the existence of such filesystems as ZFS and BTRFS is that the problem of "bit rot" should be addressed at a lower abstraction
by Koshkin 9y ago
A logical issue that I have with the existence of such filesystems as ZFS and BTRFS is that the problem of "bit rot" should be addressed at a lower abstraction level - hardware or the driver - rather than at the level that should be primarily responsible for user-visible organization of files, directories, etc.
- wtallis 9y agoIf you have a layer that knows there's another copy of the data on a different disk, that layer should be checking for bit-rot even if the individual drives have their own error correction.
- kev009 9y agoYou must not read HW errata or device drivers. It's been posited that the main push for ZFS was perceived bugs in UFS that happened to be LSI firmware bugs once they had the capabilities of ZFS to detect them.
- dragontamer 9y agoHow? Bitrot occurs because the lower-abstraction level hardware fails. When you put a Hard-Drive into storage for say 5 years, the bits may change. Even if a Hard-drive remains in constant use for 5 years... if said files or directories aren't checked and double-checked constantly, the error-correction codes may fail over time. Its a fundamentally different problem from Hard Drives that are being used constantly as say Swap. Hard Drives typically include Hamming codes or ECC bits to address typical corruption issues. ------------- The fundamental principle at hand here is as follows: to ensure integrity of files, you need to regularly check file data. Only the Filesystem would know which files were recently checked.
- Koshkin 9y agoCouldn't the drive's firmware or the driver do the same just as well (except on physical records instead of files)?
- dragontamer 9y agoFirst off, real hard drives have "SMART" data that detects (and automatically corrects) simple errors. So remember, hard drives ALREADY have a large degree of error correction built in. Its just not enough for serious data-storage purposes. The "Bit-Rot" scenario is particularly harmful to RAID5 (Minimum 3-hard drives. Two contain data, one contains "parity" that can fix any errors on the other hard drives. Then the parity is structured to be striped equally across the three drives). Modern RAID drivers can do this rather easily. The problem with "Bit Rot" is that a RAID5 drive will not rebuild itself until it detects an error. If you're reading files along, and all of a sudden... the hard drive detects an error. No problem (in the typical case), just rebuild the data from the parity. However, "Bit Rot" means that the parity bits (on the 3rd backup hard drive) have ALSO rotted away. ---------- The only way to fix this "bit-rot" error is to constantly read through your data and CONSTANTLY check for bit-rot. No hard drive is going to silently spin and hamper-performance of the system for self-verification purposes... but a Filesystem / Operating system can schedule these "Scrubs" to occur during periods of low-I/O. Which is how ZFS, and Window's ReFS work. When your computer is idle, the OS checks for bitrot. When the computer starts to work again, it pauses the "low priority" bitrot checks and serves the data. ------- ZFS doesn't quite work like Window's ReFS. ZFS simply checks for bit rot whenever a file is accessed. Every time. There are "ZFS Scrub" commands (which you can put into a cron-job) to read every file (and therefore check for bitrot).
- KaiserPro 9y ago"enterprise" storage arrays do this for you. You have a glob of storage that is presented as a block device, but actually underneath is a reas no fooling filesystem with FEC, Snapshots and all sorts of other goodies.
- lmm 9y agoYou can get a disaggregated version of what ZFS does - you can have filesystem -> LVM -> MD -> devices, where the MD layer does the checksumming at a sequence-of-bits level. It tends to work less well than ZFS though; snapshots are a lot more efficient if the low level knows which bits are which files at the high level, scrubbing can be done more efficiently if you know which files are in use and which aren't, if the parity checking wants to use idle bandwidth then it needs to know about user-requested access. Perhaps most importantly, data writes need to be async for performance, but filesystem metadata needs to be written in a way that will maintain the filesystem's invariants, so the question of when bits hit the physical disk ends up being intimately entangled with the filesystem's internals. It's really difficult to express an interface between the filesystem layer and the parity layer that lets it do the right thing.