4 ms·

It's surprising the large amount of people I have found who are incapable of conceiving the notion of what this post calls "structural error" in modern file sys

It's surprising the large amount of people I have found who are incapable of conceiving the notion of what this post calls "structural error" in modern file systems such as ZFS. They have this idea that checksums and scrubbing make ZFS invulnerable to corruption, and thus that the notion of a fsck does not make sense in ZFS. It probably makes sense for many because it fits with the fact that ZFS does not have a fsck, but that does not make that kind of corruption less real.

csdvrx 5y ago
ZFS Scrub is like a fsck in a crontab...
- chungy 5y ago
  scrub is exactly the "zfs fsck" that people claim they pine for. It is part of the file system's design philosophy that every administrative action happens on online pools, which even includes correcting errors in the file system. Making it so you don't have to take your system offline is a feature, not a bug.
  - rincebrain 5y ago
    I think the difficulty in these conversations is that people say they want a "fsck for ZFS", when often they don't mean they want "periodic sanity checks for ZFS", they mean they want something like "xfs_repair -d* for ZFS" or "extundelete for ZFS" - that is, a tool for salvaging beyond-the-pale mangling such that your pool is no longer able to be imported. * - not a perfect analogy, but I'm hard-pressed to think of good off the shelf tools for what they're looking for. I guess reiserfsck's infamously side-effect laden --rebuild-tree would probably be closest... (I am acquainted with zdb -r and import -T; neither helps you if there's not enough metadata consistent to get enough of a pool structure in memory to 'import', but one could still conceivably salvage some data in that case.)
    - XorNot 5y ago
      Technically this is the various, increasingly dangerous zfs import commands though. The point of those is that the only time a pool is truly unusable is when you can't even import it.
      - rincebrain 5y ago
        Yes, -F, -X, and -T. The issue is, I think, people are expressing a desire for a tool that can still salvage data even when you've gone through all the (128/N) -T options and found them all unhelpful.
  - siebenmann 5y ago
    Unfortunately ZFS scrubs are not as complete as fsck on a regular filesystem. ZFS scrubs only verify that checksums are intact. They don't verify that filesystem level metadata is correct (although they do verify ZFS structural metadata as part of walking everything, which isn't the same thing). For example, a ZFS scrub will not detect that a filesystem inode has certain sorts of crazy or invalid contents, or damaged ACLs. It doesn't even necessarily verify that the filesystem directory structure is correct and intact. For more on this, see https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSScrubLimitsII https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSScrubLim... (The tl;dr is that a fsck on an ordinary filesystem has to walk the directory tree to find everything. However, ZFS maintains a separate list of active inodes and a scrub can just walk over them and check the checksums of all of their data blocks. It doesn't have to, for example, read a directory's contents to find further files to scrub.)
- diegocg 5y ago
  No, it is not. ZFS scrub only checks that the checksums is valid. A scrub will not check in detail for what this post calls "structural error" (it only will find errors when doing normal walk of structures, which is not very detailed - doing a detailed file system checking is so resource intensive that some fsck implementations have a "low memory" mode in order to avoid OOM). The checksum/mirroring mechanisms cannot fix any structural error when the filesystem is doing something and finds and inconsistency. ZFS chose to not have a fsck out of pure arrogance, not because scrub is a proper substitute. ZFS developers believed that corruption bugs produced by code can be fixed by providing Bug Free Code (tm). That, and the fact that errors due to media corruption will be fixed with checksums and mirroring, made them believe that they could make fsck a thing of the past. Other modern file systems mimicking ZFS are developing a fsck, despite having scrub-like functionality. ...as I said, and your reply proves again: > It's surprising the large amount of people I have found who are incapable of conceiving the notion of what this post calls "structural error" in modern file systems such as ZFS People _really_ wants to believe ZFS has some kind of magic.
  - XorNot 5y ago
    Your post though isn't providing a single example of a "structural error" which could occur in ZFS and wouldn't be fixed by the checksums and redundancy data. What's your point here? That ZFS can't correct logic bugs in it's own implementation that would lead to structural errors? (What system could?)
    - Jenda_ 5y ago
      I don't have experience with ZFS, but in btrfs, you can get "corrupt leaf" on a FS passing scrub. This probably happens either because of bug or because the structures got corrupted (e.g. bad RAM) when they were constructed, before the checksum (of the already wrong data) was calculated. This is kind of expected behavior, as the checksums only assert that we have read from the device what was written to it previously, but they have absolutely no relation to constrains like "the blocks form a valid B-tree". Another example that I have here right now, is a directory that says the following on "ls" # l pg_stat_tmp/ ls: cannot access 'pg_stat_tmp/global.stat': No such file or directory [...] -????????? ? ? ? ? ? db_0.stat The btrfsck says stuff like "parent transid verify failed" and "ERROR: child eb corrupted". scrub finishes without errors. > That ZFS can't correct logic bugs in it's own implementation that would lead to structural errors? Again, I can't speak for ZFS, but the problem with btrfs is that for example in ext4 you have fsck that will fix such errors (sometimes losing the affected files). But in btrfs, the fsck is mostly "beta and do not use and it can't fix that, just move the data elsewhere, create the FS from scratch and move data back and hope it won't happen again".

csdvrx 5y ago

ZFS Scrub is like a fsck in a crontab...

chungy 5y ago
scrub is exactly the "zfs fsck" that people claim they pine for. It is part of the file system's design philosophy that every administrative action happens on online pools, which even includes correcting errors in the file system. Making it so you don't have to take your system offline is a feature, not a bug.
- rincebrain 5y ago
  I think the difficulty in these conversations is that people say they want a "fsck for ZFS", when often they don't mean they want "periodic sanity checks for ZFS", they mean they want something like "xfs_repair -d* for ZFS" or "extundelete for ZFS" - that is, a tool for salvaging beyond-the-pale mangling such that your pool is no longer able to be imported. * - not a perfect analogy, but I'm hard-pressed to think of good off the shelf tools for what they're looking for. I guess reiserfsck's infamously side-effect laden --rebuild-tree would probably be closest... (I am acquainted with zdb -r and import -T; neither helps you if there's not enough metadata consistent to get enough of a pool structure in memory to 'import', but one could still conceivably salvage some data in that case.)
  - XorNot 5y ago
    Technically this is the various, increasingly dangerous zfs import commands though. The point of those is that the only time a pool is truly unusable is when you can't even import it.
    - rincebrain 5y ago
      Yes, -F, -X, and -T. The issue is, I think, people are expressing a desire for a tool that can still salvage data even when you've gone through all the (128/N) -T options and found them all unhelpful.
- siebenmann 5y ago
  Unfortunately ZFS scrubs are not as complete as fsck on a regular filesystem. ZFS scrubs only verify that checksums are intact. They don't verify that filesystem level metadata is correct (although they do verify ZFS structural metadata as part of walking everything, which isn't the same thing). For example, a ZFS scrub will not detect that a filesystem inode has certain sorts of crazy or invalid contents, or damaged ACLs. It doesn't even necessarily verify that the filesystem directory structure is correct and intact. For more on this, see https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSScrubLimitsII https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSScrubLim... (The tl;dr is that a fsck on an ordinary filesystem has to walk the directory tree to find everything. However, ZFS maintains a separate list of active inodes and a scrub can just walk over them and check the checksums of all of their data blocks. It doesn't have to, for example, read a directory's contents to find further files to scrub.)
diegocg 5y ago
No, it is not. ZFS scrub only checks that the checksums is valid. A scrub will not check in detail for what this post calls "structural error" (it only will find errors when doing normal walk of structures, which is not very detailed - doing a detailed file system checking is so resource intensive that some fsck implementations have a "low memory" mode in order to avoid OOM). The checksum/mirroring mechanisms cannot fix any structural error when the filesystem is doing something and finds and inconsistency. ZFS chose to not have a fsck out of pure arrogance, not because scrub is a proper substitute. ZFS developers believed that corruption bugs produced by code can be fixed by providing Bug Free Code (tm). That, and the fact that errors due to media corruption will be fixed with checksums and mirroring, made them believe that they could make fsck a thing of the past. Other modern file systems mimicking ZFS are developing a fsck, despite having scrub-like functionality. ...as I said, and your reply proves again: > It's surprising the large amount of people I have found who are incapable of conceiving the notion of what this post calls "structural error" in modern file systems such as ZFS People _really_ wants to believe ZFS has some kind of magic.
- XorNot 5y ago
  Your post though isn't providing a single example of a "structural error" which could occur in ZFS and wouldn't be fixed by the checksums and redundancy data. What's your point here? That ZFS can't correct logic bugs in it's own implementation that would lead to structural errors? (What system could?)
  - Jenda_ 5y ago
    I don't have experience with ZFS, but in btrfs, you can get "corrupt leaf" on a FS passing scrub. This probably happens either because of bug or because the structures got corrupted (e.g. bad RAM) when they were constructed, before the checksum (of the already wrong data) was calculated. This is kind of expected behavior, as the checksums only assert that we have read from the device what was written to it previously, but they have absolutely no relation to constrains like "the blocks form a valid B-tree". Another example that I have here right now, is a directory that says the following on "ls" # l pg_stat_tmp/ ls: cannot access 'pg_stat_tmp/global.stat': No such file or directory [...] -????????? ? ? ? ? ? db_0.stat The btrfsck says stuff like "parent transid verify failed" and "ERROR: child eb corrupted". scrub finishes without errors. > That ZFS can't correct logic bugs in it's own implementation that would lead to structural errors? Again, I can't speak for ZFS, but the problem with btrfs is that for example in ext4 you have fsck that will fix such errors (sometimes losing the affected files). But in btrfs, the fsck is mostly "beta and do not use and it can't fix that, just move the data elsewhere, create the FS from scratch and move data back and hope it won't happen again".