5 ms·
It is a real bummer that there are still no content checksums available. One can work around this by doing sums and scrubs in userspace by storing mtime+hash in
by sneak 7y ago
It is a real bummer that there are still no content checksums available. One can work around this by doing sums and scrubs in userspace by storing mtime+hash in an xattr and periodically verifying and updating them. There is a utility to do this but I forgot the name.
- Hackbraten 7y agoI’m thinking of moving back from ZFS to APFS if I had such a workaround. Block- or file-level integrity is now the only thing that keeps me back.
- privateSFacct 7y agoWhat are the use cases folks have with super high reliability needs? I'm generally more worried about an SSD falling over entirely - so focus is on duplication (raid with hot spare -> offsite S3 backup).
- wang_li 7y agoFrom a user perspective it's effectively a zero cost feature. Apple should add it in.
- privateSFacct 7y agoOK - so they do a block level checksum (no error correction but it would let you know that something went wrong). Now something goes wrong with checksum. Is this a "zero cost" feature to have every app updated to handle this checksum error. What is gained? If you need reliability do something like SSD or a raid option and a backup.
- wang_li 7y agoThe apps don't have to deal with it. It's a read error. Presumably something they already deal with. There's no operational difference for the end user between a block that can't be read and a block than can be read but with corrupted data. And yes, storage would ideally be paired with another physical device to provide redundancy and a mechanism to correct the error on the erroring device.
- zeotroph 7y agoOr just pair it with iCloud and "cloud heal" the corrupted local data. Then mark the sector as bad and enjoy longer lifetime for the device.
- sneak 7y agoI just built an external storage server on a supermicro 2U with 80TB of hdd and 2TB of fast flash running ZFS and connected to my Mac via 10GE. Best of both worlds.
- zeotroph 7y agoFor a company which primarily sells consumer products it makes sense to omit hardware checksums: You just detect errors which the end user might never notice, a bit flip in a picture you never look at again might not be that bad. For anyone who has ever heard of checksums this makes APFS a no-go of course.
- privateSFacct 7y agoI guess NTFS is also a "no go" for file storage then as well. What about ext4? Oh wait - no file data checksums - so a "no go". Actually, it turns out plenty of folks have built plenty of storage on these platforms - including APFS. In particular, for APFS, apple uses ECC for all hardware, uses fletcher checksums for metadata and I believe does a copy on write approach to help handle sudden power loss events etc. It seems to be working out OK for them given the prices they charge.
- isomorphic 7y agoYour point that other filesystems get by without checksums stands, but Apple does not use ECC for all hardware. Also, we are left to wonder if their "reliable" SSDs are as reliable as checksums at the filesystem layer (or as reliable as both). I suspect the answer is yes, for the majority of applications running atop APFS (which is to say, iOS). Probably not for video assets on that Promise MPX bay (i.e. "external" non-Apple storage media).
- AlphaSite 7y agoI do t think zfs can handle memory correction without ecc either, so is no better in this case.
- zeotroph 7y agoAll the next gen, copy on write filesystems have checksums: NTFS+ (ReFS) has it, ext4+ (btrfs) has it, but HPFS++ (APFS) has not. Bummer. At least optional checksums should be supported, then watchOS does not waste time with it but a user with 10k+ mac workstation who might still uses spinning rust for bulk data storage is SOL.
- sneak 7y agohttps://github.com/ludios/bitscrub https://github.com/ludios/bitscrub This is one that does it with CRC32, which may or may not be sufficient for detecting bitrot. It is probably trivial to modify to support a stronger hash.