7 ms·
No checksumming?
by skrause 10y ago
No checksumming?
- amock 10y agoI also didn't see any mention of compression. Hopefully those are at least being worked on and just didn't make it into the docs yet.
- tambourine_man 10y agoHFS+ already supports compression and it says APFS would support nearly all of the features of HFS+, so there's hope.
- kylec 10y agoChecksumming is the thing that I want the most from a new Apple FS. I really hope they include it at some point.
- derefr 10y agoHeck, if someone's going whole-hog on file integrity, I'd really like them to support the creation of files that are stored "expanded" by a fountain-code. Rather than having to RAID-mirror every block on my disk, I'd like to be able to pick just some files and say "please store those ones slightly more redundantly, such that they're protected from bit-level disk errors—they're important."
- nisa 10y agosomething like zfs copies=n? I used this on a drive with tons of bad sectors and it worked flawless - ZFS even spreads the writes throughout the disk. However AFAIK this is a property of using merkle hash trees (just add multiple leaves) and it's probably unlikely that Apple went this way.
- cjensen 10y agoChecksumming makes no sense on modern hardware. Hard Drives and SSDs use CRCs already. When a sector goes bad on a hard drive, the firmware will mess about retrying and altering the analog amplifiers to try to get the signal back. If it gets the data back, it might "recover" by moving the data to spare sectors. Without a CRC, the firmware would have no way of differentiating between a sector read correctly and one that is read erroneously.
- prutschman 10y agoChecksumming makes sense when the checksum accompanies a pointer to another block. i.e. a directory entry says subdirectory contents are found in block 1027 /with checksum 0xdeadbeef/. If block 1027 is errantly overwritten due to a bug in the filesystem, the block driver, the DMA subsystem, or the device firmware, the FS will know as soon as it goes to fetch block 1027 that something went wrong. This is true even if the block is still internally consistent; perhaps the write was misdirected, another sector was incorrectly read, or a failure in the flash translation layer caused a newer write to be lost. If there's any redundancy to the system, whether storing the metadata in multiple places on the disk, or on different disks in an array, the FS can then check all the others and, crucially, detect which version is correct.
- ysleepy 10y agoWell it happens enough in practice, so it must come from somewhere. Maybe its the shitty SATA cable, or RAM bitflips, or the disk. I'll gladly spend 1% or whatever space overhead to notice it.
- mioelnir 10y agoSSDs use CRC to deliver their probabilistic data storage approximation. FS level checksums are, among others, for when that fails. FS level checksums also cover bitflips on your bus during the write phase (the drive will crc the already bad data) or during the read phase (you receive different data than your drive sent).
- nisa 10y agoAt least with bigger SATA disks I see regularly checksum errors on btrfs/ZFS and a rising bad sector count. The disk usually replaces the sectors on the next write but when reading it returns either zeros or wrong data. It's nice to know what files are affected from bad sectors. It's not common but common enough to have it.
- kchoudhu 10y agoCan you implement COW filesystems without some kind of checksumming? Or have I been in ZFS-land for too long?
- Freaky 10y agoYup. Nothing intrinsic to it that needs checksums. zfs set checksum=off rpool
- usefulcat 10y agoSure, just assume that there's no such thing as bit rot.
- madgar 10y agoI think you're mixing up checksumming and content-addressing.
- cmurf 10y agoYes, what's more difficult is implementing data checksumming without CoW. What you may not be used to in ZFS is a nodatacow option (per file or per filesystem) which does exist on Btrfs and it implies nodatasum (no checksums for data, metadata is still always checksummed and cow). Conversely, nodatasum does not imply nodatacow.
- wstrange 10y agoI have a theory on this regarding consumer devices: Checksumming drives a lot of support calls that would otherwise not happen. As long as the errors are in media files and not meta-data, most consumers are going to be oblivious to bit rot in their downloaded movies and photos. Enabling checksumming is going to reveal a lot of errors that would otherwise be silently ignored (spoken from experience running a ZFS media server)
- Freaky 10y agoThey could just checksum metadata, which is cheaper to store redundantly. ReFS already does this when you're running it on non-redundant storage - integrity streams are only enabled on mirrored/parity configurations.
- chongli 10y agoJust because you do checksumming doesn't mean you have to report those errors in a way that might frighten a non-technical user. How many regular Mac users sit there looking at the syslog all day?
- veidr 10y agoThat is insane! I just so strongly assumed that any new advanced filesystem would have checksumming, I read the entire document without realizing it is never mentioned. It would be unbelievable if they really didn't have checksumming. Could it just be that they haven't documented it yet? That seems weird and unlikely, but... not was weird and unlikely as APFS not having checksumming.
- Alphasite_ 10y agoIt may, or may not, have it but I very much doubt there isn't room to retrofit it where appropriate.