I've had to deal with one scenario where ZFS fsck would have been very useful, though it comes up rarely: corruption leading to valid checksums, but bad ZFS dat

I've had to deal with one scenario where ZFS fsck would have been very useful, though it comes up rarely: corruption leading to valid checksums, but bad ZFS data. In my case[1], our setup died due to some power problems, and somehow, NULL pointers got written to disk with valid checksums. Normally this wouldn't happen, but when it does, it's a PITA to debug because trying to traverse/read the disk gives a kernel fault, instead of a segfault as you might see in a user-level fsck program. This was a real pain, as we had encrypted disks, and every reboot meant going through the disk attach steps (enter password, etc etc), every time. As a result, a userspace implementation of scrubbing would be useful since in this sort of rare instance, I'd be able to probe the fsck process with a good debugger and see why it's crashing. Since it's in userspace, the fsck program can also quit more sanely, with a full report on where it found the corruption. I was able to get my data back via some ad-hoc patches, but it was an... interesting experience having to debug in kernel vs in userspace. zdb isn't a substitute for most of these things as the on-disk compression and RAIDZ sharding makes it difficult to actually see the raw data structures. Max Bruning wrote a post a while back with an on-disk data walkthrough[2], where he wrote some patches to fix this, but they haven't made their way upstream yet. Additionally, FreeBSD and Linux don't have mdb. :( [1] http://lists.freebsd.org/pipermail/freebsd-fs/2012-November/015808.html http://lists.freebsd.org/pipermail/freebsd-fs/2012-November/... [2] http://mbruning.blogspot.com/2009/12/zfs-raidz-data-walk.html http://mbruning.blogspot.com/2009/12/zfs-raidz-data-walk.htm... edit: formatting