5 ms·

The idea that you can have corruption and nothing to fix it with already sounds scary enough to me. Especially for those of us that don't have thousands of mac

The idea that you can have corruption and nothing to fix it with already sounds scary enough to me. Especially for those of us that don't have thousands of machines and can therefor be badly screwed by one issue.

dmpk2k 14y ago
Given the way ZFS works, you'll have to elaborate on what an fsck would do that ZFS doesn't automatically do already. What I find scary is silently corrupt data, something which is a problem for most other filesystems. I've seen ZFS catch and fix that error orders of magnitude more often than I've seen ZFS flake out. If we're talking risk analysis, I feel you're worried about a mouse in the corner, while a starved tiger is hungrily licking its chops while staring at you.
- XorNot 14y ago
  Just look at the recent KDE git disaster. A lot of things went wrong there, but fundamentally the issue was ext4 silently returning bad data. The thing about fsck-like recovery tools is you need to have a failure mode in mind when you write them. ZFS can fix most of those types of errors thanks to the checksums and on-disk redundancy on the fly. Or at least tell you that something is now going wrong and which files are affected.
- wereHamster 14y ago
  I think it's unfortunate that Sun called it 'scrub' instead of 'fsck'. Because the two are largely equivalent in functionality (to the extents supported by the filesystem). Both fix filesystem corruption if they can. Just scrub can be run while the filesystem is mounted, whereas the traditional fsck must be run when the filesystem is offline. However, scrub does not make ZFS perfect. There are still ways the filesystem can become corrupted without scrub noticing. Or corrupted in a way so that ZFS fails to recover from, even though recovering would be dead simple. The attitude of the ZFS developers only works in the enterprise market: Your data is safe (checksummed, scrubbed, replicated using RAID-Z), but if a bit flips in the superblock just restore from your backup, because we won't provide tools to recover from that.
  - dmpk2k 14y ago
    While I concur with most of your points, I must point out that there are four uberblocks, not one; a flipped bit will not impair the pool. Let's argue that bits flip in all four uberblocks though, then ZFS will use the previous valid commit, which also has four uberblocks (ZFS is a CoW FS). And so on backwards for quite a few transactions. All these uberblocks are spread out at the beginning and end of each disk. ZFS has a policy that the more important the data (and the top of the tree is most important), the more copies there are, although a user can also define how many duplicates there should be at the leaves of the tree. Basically, you'd need a very trashed disk to render an entire pool dead. You're not going to recover from that, regardless of filesystem.
    - wereHamster 14y ago
      Well, ZFS clearly didn't use the extra copies, nor did it try to use the previous uberbocks. Otherwise it would've been able to mount the filesystem, don't you think? And I disagree that the disk was very trashed, if all it took was to invalidate a few uburblocks to get the filesystem mounted again.
      - dmpk2k 14y ago
        Did you use zpool import -F (are you sure)? I'm surprised that'd be necessary though, given my experiences. How long ago was this?
glabifrons 14y ago
As others have stated, if the current state has corruption, you roll it back. As I posted in response to another post above, this is as easy as "zpool clear -F $POOLNAME". The fact that the vast majority of other filesystems have no way to detect silent corruption of data (only metadata inconsistencies) is far more frightening to me. Here is an nice article written by someone who discovered just how unreliable disks are, after switching to ZFS (because other filesystems couldn't detect the corruption). Quite an eye-opener. http://www.oracle.com/technetwork/systems/opensolaris/data-rot-jsp-135610.html http://www.oracle.com/technetwork/systems/opensolaris/data-r... If you use chrome and are getting the same error as I am, here's the google cache link (Firefox will load it): http://webcache.googleusercontent.com/search?q=cache:caEwhGDIoGIJ:www.oracle.com/technetwork/systems/opensolaris/data-rot-jsp-135610.html+&cd=2&hl=en&ct=clnk&gl=us http://webcache.googleusercontent.com/search?q=cache:caEwhGD... The article references sources of studies of hard disk corruption, if you'd want something with even more detail and statistics: http://research.cs.wisc.edu/wind/Publications/pointer-dsn08.pdf http://research.cs.wisc.edu/wind/Publications/pointer-dsn08.... http://static.usenix.org/events/fast08/tech/full_papers/bairavasundaram/bairavasundaram.pdf http://static.usenix.org/events/fast08/tech/full_papers/bair...

dmpk2k 14y ago

Given the way ZFS works, you'll have to elaborate on what an fsck would do that ZFS doesn't automatically do already. What I find scary is silently corrupt data, something which is a problem for most other filesystems. I've seen ZFS catch and fix that error orders of magnitude more often than I've seen ZFS flake out. If we're talking risk analysis, I feel you're worried about a mouse in the corner, while a starved tiger is hungrily licking its chops while staring at you.

XorNot 14y ago
Just look at the recent KDE git disaster. A lot of things went wrong there, but fundamentally the issue was ext4 silently returning bad data. The thing about fsck-like recovery tools is you need to have a failure mode in mind when you write them. ZFS can fix most of those types of errors thanks to the checksums and on-disk redundancy on the fly. Or at least tell you that something is now going wrong and which files are affected.
wereHamster 14y ago
I think it's unfortunate that Sun called it 'scrub' instead of 'fsck'. Because the two are largely equivalent in functionality (to the extents supported by the filesystem). Both fix filesystem corruption if they can. Just scrub can be run while the filesystem is mounted, whereas the traditional fsck must be run when the filesystem is offline. However, scrub does not make ZFS perfect. There are still ways the filesystem can become corrupted without scrub noticing. Or corrupted in a way so that ZFS fails to recover from, even though recovering would be dead simple. The attitude of the ZFS developers only works in the enterprise market: Your data is safe (checksummed, scrubbed, replicated using RAID-Z), but if a bit flips in the superblock just restore from your backup, because we won't provide tools to recover from that.
- dmpk2k 14y ago
  While I concur with most of your points, I must point out that there are four uberblocks, not one; a flipped bit will not impair the pool. Let's argue that bits flip in all four uberblocks though, then ZFS will use the previous valid commit, which also has four uberblocks (ZFS is a CoW FS). And so on backwards for quite a few transactions. All these uberblocks are spread out at the beginning and end of each disk. ZFS has a policy that the more important the data (and the top of the tree is most important), the more copies there are, although a user can also define how many duplicates there should be at the leaves of the tree. Basically, you'd need a very trashed disk to render an entire pool dead. You're not going to recover from that, regardless of filesystem.
  - wereHamster 14y ago
    Well, ZFS clearly didn't use the extra copies, nor did it try to use the previous uberbocks. Otherwise it would've been able to mount the filesystem, don't you think? And I disagree that the disk was very trashed, if all it took was to invalidate a few uburblocks to get the filesystem mounted again.
    - dmpk2k 14y ago
      Did you use zpool import -F (are you sure)? I'm surprised that'd be necessary though, given my experiences. How long ago was this?

glabifrons 14y ago

As others have stated, if the current state has corruption, you roll it back. As I posted in response to another post above, this is as easy as "zpool clear -F $POOLNAME". The fact that the vast majority of other filesystems have no way to detect silent corruption of data (only metadata inconsistencies) is far more frightening to me. Here is an nice article written by someone who discovered just how unreliable disks are, after switching to ZFS (because other filesystems couldn't detect the corruption). Quite an eye-opener. http://www.oracle.com/technetwork/systems/opensolaris/data-rot-jsp-135610.html http://www.oracle.com/technetwork/systems/opensolaris/data-r... If you use chrome and are getting the same error as I am, here's the google cache link (Firefox will load it): http://webcache.googleusercontent.com/search?q=cache:caEwhGDIoGIJ:www.oracle.com/technetwork/systems/opensolaris/data-rot-jsp-135610.html+&cd=2&hl=en&ct=clnk&gl=us http://webcache.googleusercontent.com/search?q=cache:caEwhGD... The article references sources of studies of hard disk corruption, if you'd want something with even more detail and statistics: http://research.cs.wisc.edu/wind/Publications/pointer-dsn08.pdf http://research.cs.wisc.edu/wind/Publications/pointer-dsn08.... http://static.usenix.org/events/fast08/tech/full_papers/bairavasundaram/bairavasundaram.pdf http://static.usenix.org/events/fast08/tech/full_papers/bair...