5 ms·

The article is correct but it downplays an important limitation of ZFS scrubs when it talks about how they're different from fsck and chkdsk. As the article say

by thatcks 9mo ago

The article is correct but it downplays an important limitation of ZFS scrubs when it talks about how they're different from fsck and chkdsk. As the article says (in different words), ZFS scrubs do not check filesystem objects for correctness and consistency; it only checks that they have the expected checksum and so have not become corrupted due to disk errors or other problems. Unfortunately it's possible for ZFS bugs and issues to give you filesystem objects that have problems, and as it stands today ZFS doesn't have anything that either checks or corrects these. Sometimes you find them through incorrect results; sometimes you discover they exist through ZFS assertion failures triggering kernel panics. (We run ZFS in production and have not been hit by these issues, at least not that we know about. But I know of some historical ZFS bugs in this area and mysterious issues that AFAIK have never been fully diagnosed.)

mustache_kimono 9mo ago
"Scrubs differ significantly from traditional filesystem checks. Tools such as fsck or chkdsk examine logical structures and attempt to repair inconsistencies related to directory trees, allocation maps, reference counts, and other metadata relationships. ZFS does not need to perform these operations during normal scrubs because its transactional design ensures metadata consistency. Every transaction group moves the filesystem from one valid state to another. The scrub verifies the correctness of the data and metadata at the block level, not logical relationships." > ZFS scrubs do not check filesystem objects for correctness and consistency; it only checks that they have the expected checksum and so have not become corrupted due to disk errors or other problems A scrub literally reads the object from disk. And, for each block, the checksums are read up the tree. The object is therefore guaranteed to be correct and consistent at least re: the tree of blocks written. > Unfortunately it's possible for ZFS bugs and issues to give you filesystem objects that have problems Can you give a more concrete example of what you mean? It sounds like you have some experience with ZFS, but "ZFS doesn't have an fsck" is also some truly ancient FUD, so you will forgive my skepticism. I'm willing to believe that you request an object and ZFS cannot return that object because of ... a checksum error or a read error in a single disk configuration, but what I have never seen is a scrub that indicates everything is fine, and then reads which don't return an object (because scrubs are just reads themselves?). Now, are things like pool metadata corruption possible in ZFS? Yes, certainly. I'm just not sure fsck would or could help you out of the same jam if you were using XFS or ext4. AFAIK fsck may repair inconsistencies but I'm not sure it can repair metadata any better than ZFS can?
- ori_b 9mo ago
  Imagine a race condition that writes a file node where a directory node should be. You have a valid object with a valid checksum, but it's hooked into the wrong place in your data structure.
  - mustache_kimono 9mo ago
    > Imagine a race condition that writes a file node where a directory node should be. You have a valid object with a valid checksum, but it's hooked into the wrong place in your data structure. A few things: 1) Is this an actual ZFS issue you encountered or is this a hypothetical? 2) And -- you don't imagine this would be discovered during a scrub? Why not? 3) But -- you do imagine it would be discovered and repaired by an fsck instead? Why so? 4) If so, wouldn't this just be a bug, like a fsck, not some fundamental limitation of the system? FWIW I've never seen anything like this. I have seen Linux plus a flaky ALPM implementation drop reads and writes. I have seen ZFS notice at the very same moment when the power dropped via errors in `zpool status`. I do wonder if ext4's fsck or XFS's fsck does the same when someone who didn't know any better (like me!) sets the power management policy to "min_power" or "med_power_with_dipm".
    - ori_b 9mo ago
      Here's an example: https://www.illumos.org/issues/17734 https://www.illumos.org/issues/17734. But it would not be discovered by a scrub because the hashes are valid. Scrubs check hashes, not structure. It would be discovered by a fsck because the structure is invalid. Fscks check structure, not hashes. They are two different tools, with two different uses.
      - mustache_kimono 9mo ago
        > Scrubs check hashes, not structure. How is the structure not valid here? Can you explain to us how an fsck would discover this bug (show an example where an fsck fixed a similar bug) but ZFS could never? The point I take contention with is that missing an fsck is a problem for ZFS, so more specifically can you answer my 4th Q: >> 4) If so, wouldn't this just be a bug, like (a bug in) fsck, not some fundamental limitation of the system? So -- is it possible an fsck might discover an inconsistency ZFS couldn't? Sure. Would this be a fundamental flaw of ZFS, which requires an fsck, instead of merely a bug? I'm less sure. You do seem to at least understand my general contention with the parent's point. However, the parent is also making a specific claim about a bug which would be extraordinary. Parent's claim is this is a bug which a scrub, which is just a read, wouldn't see, but a subsequent read would reveal. So -- is it possible an fsck might discover this specific kind of extraordinary bug in ZFS, after a scrub had already read back the data? Of that I'm highly dubious.
wereHamster 9mo ago
A loooong time age (OpenSolaris days) I had a system that had corrupted its zfs. No fsck was available because the developers claimed (maybe still do) that it's unnecessary. I had to poke around the raw device (with dd and such) to restore the primary superblock with one of the copies (that zfs keeps in different locations on the device). So clearly the zfs devs thought about the possibility of a corrupt superblock, but didn't feel the need to provide a tool to compare the superblocks and restore one from the other copies. That was the point when I stopped trusting zfs. Such arrogance…
- fvv 9mo ago
  it's still the case even with now openzfs ? what do you trust now ?
- barrkel 9mo ago
  That's a fine fit of pique - and I once had an awkward file on one of my zfs pools, about three pools ago - but how does it leave you better off, if you want what zfs offers?
  - Dylan16807 9mo ago
    > That's a fine fit of pique So you're rejecting a story about a real bug because...? > how does it leave you better off That's a really mercenary way to look at learning about your tools. But presumably they take smaller risks around zfs systems than they otherwise would.
- throw0101a 9mo ago
  > So clearly the zfs devs thought about the possibility of a corrupt superblock, but didn't feel the need to provide a tool to compare the superblocks and restore one from the other copies. This mailing list post from 2008 talks about using zdb(8) to mark mark certain uberblocks an invalid so another one would be used: * https://zfs-discuss.opensolaris.narkive.com/Tx4FaUMv/need-help-invalidating-uberblock https://zfs-discuss.opensolaris.narkive.com/Tx4FaUMv/need-he... ZDB = ZFS debugger. It's been there since the original Solaris release of ZFS. > That was the point when I stopped trusting zfs. As opposed to trusting other file systems and volume managers, which do not have checksums, and so you wouldn't even know about the problem in the first place?
deleted 9mo ago
[deleted]
p_l 9mo ago
In my experience[1], the fsck for given filesystem will happily replicate the errors, sometimes in random ways, because often it cannot figure which road to take in face of inconsistency. If anything, OpenZFS built upon that by now documenting the previously deeply hidden option to "rewind" ZFS uberblock if the breakage is recent enough. [1] I've seen combination of ubuntu bug in packaging (of grub, of all things) and e2fsck nearly wipe a small company from existence, because e2fsck ended up trusting the data it got from superblock when it was not consistent.
- phil21 9mo ago
  > If anything, OpenZFS built upon that by now documenting the previously deeply hidden option to "rewind" ZFS uberblock if the breakage is recent enough. One of the most "wizardry" moments in my career I've personally witnessed was a deep-level ZFS expert (core OpenZFS developer) we had on retainer come in during a sev0 emergency and rapidly diagnose/rollback a very broken ZFS filesystem to a previous version from a few hours before the incident happened. This was entirely user error (an admin connected redundant ZFS "heads" to the same JBOD in an incorrect manner so both thought they were primary and both wrote to the disks) that we caught more or less immediately so the damage was somewhat limited. At the time we thought we were screwed and would have to restore from the previous days backup with a multi-day (at best) time to repair. This was on illumos a few years after the Solaris fork, so I don't think this feature was documented at the time. It certainly was a surprise to me, even though I knew that "in theory" such capability existed. The CLI incantations though were pure wizardry level stuff, especially watching it in real time with terminal sharing with someone who very much knew what they were doing.

by thatcks 9mo ago

mustache_kimono 9mo ago

"Scrubs differ significantly from traditional filesystem checks. Tools such as fsck or chkdsk examine logical structures and attempt to repair inconsistencies related to directory trees, allocation maps, reference counts, and other metadata relationships. ZFS does not need to perform these operations during normal scrubs because its transactional design ensures metadata consistency. Every transaction group moves the filesystem from one valid state to another. The scrub verifies the correctness of the data and metadata at the block level, not logical relationships." > ZFS scrubs do not check filesystem objects for correctness and consistency; it only checks that they have the expected checksum and so have not become corrupted due to disk errors or other problems A scrub literally reads the object from disk. And, for each block, the checksums are read up the tree. The object is therefore guaranteed to be correct and consistent at least re: the tree of blocks written. > Unfortunately it's possible for ZFS bugs and issues to give you filesystem objects that have problems Can you give a more concrete example of what you mean? It sounds like you have some experience with ZFS, but "ZFS doesn't have an fsck" is also some truly ancient FUD, so you will forgive my skepticism. I'm willing to believe that you request an object and ZFS cannot return that object because of ... a checksum error or a read error in a single disk configuration, but what I have never seen is a scrub that indicates everything is fine, and then reads which don't return an object (because scrubs are just reads themselves?). Now, are things like pool metadata corruption possible in ZFS? Yes, certainly. I'm just not sure fsck would or could help you out of the same jam if you were using XFS or ext4. AFAIK fsck may repair inconsistencies but I'm not sure it can repair metadata any better than ZFS can?

ori_b 9mo ago
Imagine a race condition that writes a file node where a directory node should be. You have a valid object with a valid checksum, but it's hooked into the wrong place in your data structure.
- mustache_kimono 9mo ago
  > Imagine a race condition that writes a file node where a directory node should be. You have a valid object with a valid checksum, but it's hooked into the wrong place in your data structure. A few things: 1) Is this an actual ZFS issue you encountered or is this a hypothetical? 2) And -- you don't imagine this would be discovered during a scrub? Why not? 3) But -- you do imagine it would be discovered and repaired by an fsck instead? Why so? 4) If so, wouldn't this just be a bug, like a fsck, not some fundamental limitation of the system? FWIW I've never seen anything like this. I have seen Linux plus a flaky ALPM implementation drop reads and writes. I have seen ZFS notice at the very same moment when the power dropped via errors in `zpool status`. I do wonder if ext4's fsck or XFS's fsck does the same when someone who didn't know any better (like me!) sets the power management policy to "min_power" or "med_power_with_dipm".
  - ori_b 9mo ago
    Here's an example: https://www.illumos.org/issues/17734 https://www.illumos.org/issues/17734. But it would not be discovered by a scrub because the hashes are valid. Scrubs check hashes, not structure. It would be discovered by a fsck because the structure is invalid. Fscks check structure, not hashes. They are two different tools, with two different uses.
    - mustache_kimono 9mo ago
      > Scrubs check hashes, not structure. How is the structure not valid here? Can you explain to us how an fsck would discover this bug (show an example where an fsck fixed a similar bug) but ZFS could never? The point I take contention with is that missing an fsck is a problem for ZFS, so more specifically can you answer my 4th Q: >> 4) If so, wouldn't this just be a bug, like (a bug in) fsck, not some fundamental limitation of the system? So -- is it possible an fsck might discover an inconsistency ZFS couldn't? Sure. Would this be a fundamental flaw of ZFS, which requires an fsck, instead of merely a bug? I'm less sure. You do seem to at least understand my general contention with the parent's point. However, the parent is also making a specific claim about a bug which would be extraordinary. Parent's claim is this is a bug which a scrub, which is just a read, wouldn't see, but a subsequent read would reveal. So -- is it possible an fsck might discover this specific kind of extraordinary bug in ZFS, after a scrub had already read back the data? Of that I'm highly dubious.

wereHamster 9mo ago

A loooong time age (OpenSolaris days) I had a system that had corrupted its zfs. No fsck was available because the developers claimed (maybe still do) that it's unnecessary. I had to poke around the raw device (with dd and such) to restore the primary superblock with one of the copies (that zfs keeps in different locations on the device). So clearly the zfs devs thought about the possibility of a corrupt superblock, but didn't feel the need to provide a tool to compare the superblocks and restore one from the other copies. That was the point when I stopped trusting zfs. Such arrogance…

fvv 9mo ago
it's still the case even with now openzfs ? what do you trust now ?
barrkel 9mo ago
That's a fine fit of pique - and I once had an awkward file on one of my zfs pools, about three pools ago - but how does it leave you better off, if you want what zfs offers?
- Dylan16807 9mo ago
  > That's a fine fit of pique So you're rejecting a story about a real bug because...? > how does it leave you better off That's a really mercenary way to look at learning about your tools. But presumably they take smaller risks around zfs systems than they otherwise would.
throw0101a 9mo ago
> So clearly the zfs devs thought about the possibility of a corrupt superblock, but didn't feel the need to provide a tool to compare the superblocks and restore one from the other copies. This mailing list post from 2008 talks about using zdb(8) to mark mark certain uberblocks an invalid so another one would be used: * https://zfs-discuss.opensolaris.narkive.com/Tx4FaUMv/need-help-invalidating-uberblock https://zfs-discuss.opensolaris.narkive.com/Tx4FaUMv/need-he... ZDB = ZFS debugger. It's been there since the original Solaris release of ZFS. > That was the point when I stopped trusting zfs. As opposed to trusting other file systems and volume managers, which do not have checksums, and so you wouldn't even know about the problem in the first place?

deleted 9mo ago

[deleted]

p_l 9mo ago

In my experience[1], the fsck for given filesystem will happily replicate the errors, sometimes in random ways, because often it cannot figure which road to take in face of inconsistency. If anything, OpenZFS built upon that by now documenting the previously deeply hidden option to "rewind" ZFS uberblock if the breakage is recent enough. [1] I've seen combination of ubuntu bug in packaging (of grub, of all things) and e2fsck nearly wipe a small company from existence, because e2fsck ended up trusting the data it got from superblock when it was not consistent.

phil21 9mo ago
> If anything, OpenZFS built upon that by now documenting the previously deeply hidden option to "rewind" ZFS uberblock if the breakage is recent enough. One of the most "wizardry" moments in my career I've personally witnessed was a deep-level ZFS expert (core OpenZFS developer) we had on retainer come in during a sev0 emergency and rapidly diagnose/rollback a very broken ZFS filesystem to a previous version from a few hours before the incident happened. This was entirely user error (an admin connected redundant ZFS "heads" to the same JBOD in an incorrect manner so both thought they were primary and both wrote to the disks) that we caught more or less immediately so the damage was somewhat limited. At the time we thought we were screwed and would have to restore from the previous days backup with a multi-day (at best) time to repair. This was on illumos a few years after the Solaris fork, so I don't think this feature was documented at the time. It certainly was a surprise to me, even though I knew that "in theory" such capability existed. The CLI incantations though were pure wizardry level stuff, especially watching it in real time with terminal sharing with someone who very much knew what they were doing.