6 ms·

Never mind all the promised wondrous features of ZFS, what are the recovery / fsck tools like? If the data recovery tools aren't mature, reliable and useful, th

Never mind all the promised wondrous features of ZFS, what are the recovery / fsck tools like? If the data recovery tools aren't mature, reliable and useful, then I'd advise anyone to stay away from a filesystem, no matter how world-ready the underlying FS code is. I'm probably biased; but I got screwed using ReiserFS. It was fast and great, but one day something went wrong and then I found that the reiserfsck program was practically useless. Very little work had been done on it, so any small inconsistencies in the FS meant your data was toast. Sure, keep backups and all that, but just be aware how important a good fsck tool is.

jiggy2011 14y ago
My understanding is that ZFS does a lot of healing itself on the fly. When you access a block, the block is checked against a checksum stored elsewhere on disk. If the block is bad, it is restored from a replica (so it's advisable to have 2 disks) while the original is healed. On writes ZFS also checks that each written block is valid before committing the new block to the filesystem. In the event of a bad write the new block is not committed and the old block remains, blocks are written originally to a separate area of the disk.
- joosters 14y ago
  Yes, and those are good features. But there's still plenty of room for errors. Bugs in the code, bad interactions with hardware, etc. Those are areas where ZFS is likely to be just as vulnerable as other file systems. More so, in fact, given the code size/complexity and its (relative) immaturity. I suspect (but I have no data!) that the majority of times that users have file system corruption and need to run fsck is due to bad software behaviour, and not failing disks.
  - jiggy2011 14y ago
    ZFS itself has been around for a while, was originally part of Solaris IIRC. It's using it on Linux that's new. If you have a bad software write, ZFS should be able to detect that in the same way it would a hardware error since the checksums won't match. Also worth noting that the checksums are themselves checksummed in a merkle tree. It also has a tool called "scrub" which can check data integrity without unmounting the disk. Of course if your filesystem has buggy code for checksumming/repair then you are boned whatever.
  - dmpk2k 14y ago
    Disk lie. A lot. Ask anybody who has the misfortune to work on a filesystem for any length of time. One thing to keep in mind is that ZFS is more complex than your typical filesystem, but it's also one of the few filesystems that's effectively a cryptographically self-validating tree. You're a lot more likely to have a disk lie to you than ZFS flake out.
    - tjoff 14y ago
      And the whole point of RAID is to combat that... But since there is no decent recovery tools for ZFS your entire FS might be toast if the wrong bits are turned. Yes ZFS is quite resilient against corruption but when it bites, which has happened plenty of times in the past, you are truly screwed. The philosophy behind ZFS and file system recovery has always been that it is too inconvenient. It is much better to restore from tape backups so why make any effort on recovery? Thing is, most consumers don't have tape backups... Because of that, ZFS is in the very vast majority of cases a bad choice for home users, especially under linux where it can hardly be considered mature. BRTFS just isn't ready yet. As a home user you can't even get a decent COW FS with decent snapshotting or checksums. That is the sad state we are in and will be in for many years to come.
      - jiggy2011 14y ago
        I don't quite understand what corruption scenarios fsck on Ext4 would save you from that ZFS wouldn't? Most data on consumer's computers is 1 disk failure away from loss anyway, hence the popularity of cloud syncing services.
      - X-Istence 14y ago
        Please tell me how my entire FS will be toast if the wrong bits are turned, especially when zraid2 is really the only valid solution that should be used. I've been using ZFS for years now, i've had disks go bad, I've replaced them. ZFS has protected me from data loss better than anything else. In the past when I've had the disk lie to me, ZFS was able to tell me what files were corrupted and thus had invalid data. I'm going to need to throw a [citation needed] on your post.
wcoenen 14y ago
See Checking ZFS File System Integrity[1] in the ZFS administration guide. [1] http://docs.oracle.com/cd/E19082-01/817-2271/gbbwa/ http://docs.oracle.com/cd/E19082-01/817-2271/gbbwa/
- joosters 14y ago
  A useful link. But it is unclear just how much a 'zpool scrub' checks. Sure, it is double checking the block checksums to ensure that your file contents haven't become corrupted. But how much checking does it do to the ZFS structures themselves? At first glance, it certainly seems to depend upon some high-level ZFS data in order to start. A command like 'zpool scrub pool-name' still needs to navigate the ZFS pool data on disk in order to locate the named pool.
  - dmpk2k 14y ago
    But how much checking does it do to the ZFS structures themselves? They're validated as well. Everything has a cryptographic checksum that's stored in the parent block, starting at the data and working all the way up to the top of the tree. Furthermore, the higher up the tree you go, the more redundant copies there are. The top of the tree has four copies, if I recall. This ignores support for mirroring and striping, which further improves data redundancy. But wait, there's more! Those four blocks aren't overwritten. ZFS is a copy-on-write filesystem (data and metadata) which behaves a lot like the persistent data structures that Clojure hackers are so fond of, so if the newest writes do not validate, it'll roll back to the newest valid commit of the tree. That's a nice way of saying that you're guessing given your experience with other filesystems. ZFS was a genuinely revolutionary filesystem, and doesn't behave like other filesystems, the sole OSS exception being BTRFS. Read up on it a bit, you'll find something interesting. :)
  - X-Istence 14y ago
    The pool data is stored on disk, but when a pool is already available on the system that information is cached in memory and on disk in a zpool.cache file. That data may be stale or wrong, you can in that case export the zpool and re-import it (or if it is missing, re-import with -f to force it). It then reads the data about what zpool a disk belongs to from disk. That data is itself stored in multiple locations so that it is unlikely that all of those are corrupted, after importing the disks you can run scrub. If the pool metadata itself is corrupted, generally you can roll-back to a previous time when the data is not corrupted. This document: http://docs.oracle.com/cd/E19082-01/817-2271/6mhupg6qg/index.html http://docs.oracle.com/cd/E19082-01/817-2271/6mhupg6qg/index... describes fairly well what all the options along the way are. None of the failure modes described would be any better if there was a fsck tool available... in all file systems it is going to cause dataloss.
wereHamster 14y ago
There is no fsck for ZFS. Because the ZFS developers at Sun/Oracle believe that ZFS is robust enough to never corrupt (untrue) and in case it does you should have a backup. I got screwed by ZFS, and I was able to recover the filesystem by learning its on-disk structure and fixing it directly on-disk. Something that a decent fsck tool could do. But no, ZFS never breaks. Go figure.
- dmpk2k 14y ago
  My employer has thousands of large machines with ZFS on them. We've seen corruption happen once, five years ago. Maybe we've just been really lucky.
  - onemorepassword 14y ago
    The idea that you can have corruption and nothing to fix it with already sounds scary enough to me. Especially for those of us that don't have thousands of machines and can therefor be badly screwed by one issue.
    - dmpk2k 14y ago
      Given the way ZFS works, you'll have to elaborate on what an fsck would do that ZFS doesn't automatically do already. What I find scary is silently corrupt data, something which is a problem for most other filesystems. I've seen ZFS catch and fix that error orders of magnitude more often than I've seen ZFS flake out. If we're talking risk analysis, I feel you're worried about a mouse in the corner, while a starved tiger is hungrily licking its chops while staring at you.
      - XorNot 14y ago
        Just look at the recent KDE git disaster. A lot of things went wrong there, but fundamentally the issue was ext4 silently returning bad data. The thing about fsck-like recovery tools is you need to have a failure mode in mind when you write them. ZFS can fix most of those types of errors thanks to the checksums and on-disk redundancy on the fly. Or at least tell you that something is now going wrong and which files are affected.
antonios 14y ago
ZFS needs no fsck. Period. http://www.c0t0d0s0.org/archives/6071-No,-ZFS-really-doesnt-need-a-fsck.html http://www.c0t0d0s0.org/archives/6071-No,-ZFS-really-doesnt-...
- tjoff 14y ago
  Actually. It does. Unless you assume that all hardware is perfect and that no bugs exists in ZFS (have been plenty of those in the past that just makes the whole FS completely worthless). Nut no, ZFS doesn't need fsck. Because, we don't want it to need one. Oh, and we don't want to spend the resources developing one. The only reason ZFS doesn't have a fsck tool is because in the enterprise world it doesn't need one. When it is needed you just restore from tape instead. It is that simple.
  - reeenobo 14y ago
    Actually. It. Does. Not.

jiggy2011 14y ago

My understanding is that ZFS does a lot of healing itself on the fly. When you access a block, the block is checked against a checksum stored elsewhere on disk. If the block is bad, it is restored from a replica (so it's advisable to have 2 disks) while the original is healed. On writes ZFS also checks that each written block is valid before committing the new block to the filesystem. In the event of a bad write the new block is not committed and the old block remains, blocks are written originally to a separate area of the disk.

joosters 14y ago
Yes, and those are good features. But there's still plenty of room for errors. Bugs in the code, bad interactions with hardware, etc. Those are areas where ZFS is likely to be just as vulnerable as other file systems. More so, in fact, given the code size/complexity and its (relative) immaturity. I suspect (but I have no data!) that the majority of times that users have file system corruption and need to run fsck is due to bad software behaviour, and not failing disks.
- jiggy2011 14y ago
  ZFS itself has been around for a while, was originally part of Solaris IIRC. It's using it on Linux that's new. If you have a bad software write, ZFS should be able to detect that in the same way it would a hardware error since the checksums won't match. Also worth noting that the checksums are themselves checksummed in a merkle tree. It also has a tool called "scrub" which can check data integrity without unmounting the disk. Of course if your filesystem has buggy code for checksumming/repair then you are boned whatever.
- dmpk2k 14y ago
  Disk lie. A lot. Ask anybody who has the misfortune to work on a filesystem for any length of time. One thing to keep in mind is that ZFS is more complex than your typical filesystem, but it's also one of the few filesystems that's effectively a cryptographically self-validating tree. You're a lot more likely to have a disk lie to you than ZFS flake out.
  - tjoff 14y ago
    And the whole point of RAID is to combat that... But since there is no decent recovery tools for ZFS your entire FS might be toast if the wrong bits are turned. Yes ZFS is quite resilient against corruption but when it bites, which has happened plenty of times in the past, you are truly screwed. The philosophy behind ZFS and file system recovery has always been that it is too inconvenient. It is much better to restore from tape backups so why make any effort on recovery? Thing is, most consumers don't have tape backups... Because of that, ZFS is in the very vast majority of cases a bad choice for home users, especially under linux where it can hardly be considered mature. BRTFS just isn't ready yet. As a home user you can't even get a decent COW FS with decent snapshotting or checksums. That is the sad state we are in and will be in for many years to come.
    - jiggy2011 14y ago
      I don't quite understand what corruption scenarios fsck on Ext4 would save you from that ZFS wouldn't? Most data on consumer's computers is 1 disk failure away from loss anyway, hence the popularity of cloud syncing services.
    - X-Istence 14y ago
      Please tell me how my entire FS will be toast if the wrong bits are turned, especially when zraid2 is really the only valid solution that should be used. I've been using ZFS for years now, i've had disks go bad, I've replaced them. ZFS has protected me from data loss better than anything else. In the past when I've had the disk lie to me, ZFS was able to tell me what files were corrupted and thus had invalid data. I'm going to need to throw a [citation needed] on your post.

wcoenen 14y ago

See Checking ZFS File System Integrity[1] in the ZFS administration guide. [1] http://docs.oracle.com/cd/E19082-01/817-2271/gbbwa/ http://docs.oracle.com/cd/E19082-01/817-2271/gbbwa/

joosters 14y ago
A useful link. But it is unclear just how much a 'zpool scrub' checks. Sure, it is double checking the block checksums to ensure that your file contents haven't become corrupted. But how much checking does it do to the ZFS structures themselves? At first glance, it certainly seems to depend upon some high-level ZFS data in order to start. A command like 'zpool scrub pool-name' still needs to navigate the ZFS pool data on disk in order to locate the named pool.
- dmpk2k 14y ago
  But how much checking does it do to the ZFS structures themselves? They're validated as well. Everything has a cryptographic checksum that's stored in the parent block, starting at the data and working all the way up to the top of the tree. Furthermore, the higher up the tree you go, the more redundant copies there are. The top of the tree has four copies, if I recall. This ignores support for mirroring and striping, which further improves data redundancy. But wait, there's more! Those four blocks aren't overwritten. ZFS is a copy-on-write filesystem (data and metadata) which behaves a lot like the persistent data structures that Clojure hackers are so fond of, so if the newest writes do not validate, it'll roll back to the newest valid commit of the tree. That's a nice way of saying that you're guessing given your experience with other filesystems. ZFS was a genuinely revolutionary filesystem, and doesn't behave like other filesystems, the sole OSS exception being BTRFS. Read up on it a bit, you'll find something interesting. :)
- X-Istence 14y ago
  The pool data is stored on disk, but when a pool is already available on the system that information is cached in memory and on disk in a zpool.cache file. That data may be stale or wrong, you can in that case export the zpool and re-import it (or if it is missing, re-import with -f to force it). It then reads the data about what zpool a disk belongs to from disk. That data is itself stored in multiple locations so that it is unlikely that all of those are corrupted, after importing the disks you can run scrub. If the pool metadata itself is corrupted, generally you can roll-back to a previous time when the data is not corrupted. This document: http://docs.oracle.com/cd/E19082-01/817-2271/6mhupg6qg/index.html http://docs.oracle.com/cd/E19082-01/817-2271/6mhupg6qg/index... describes fairly well what all the options along the way are. None of the failure modes described would be any better if there was a fsck tool available... in all file systems it is going to cause dataloss.

wereHamster 14y ago

There is no fsck for ZFS. Because the ZFS developers at Sun/Oracle believe that ZFS is robust enough to never corrupt (untrue) and in case it does you should have a backup. I got screwed by ZFS, and I was able to recover the filesystem by learning its on-disk structure and fixing it directly on-disk. Something that a decent fsck tool could do. But no, ZFS never breaks. Go figure.

dmpk2k 14y ago
My employer has thousands of large machines with ZFS on them. We've seen corruption happen once, five years ago. Maybe we've just been really lucky.
- onemorepassword 14y ago
  The idea that you can have corruption and nothing to fix it with already sounds scary enough to me. Especially for those of us that don't have thousands of machines and can therefor be badly screwed by one issue.
  - dmpk2k 14y ago
    Given the way ZFS works, you'll have to elaborate on what an fsck would do that ZFS doesn't automatically do already. What I find scary is silently corrupt data, something which is a problem for most other filesystems. I've seen ZFS catch and fix that error orders of magnitude more often than I've seen ZFS flake out. If we're talking risk analysis, I feel you're worried about a mouse in the corner, while a starved tiger is hungrily licking its chops while staring at you.
    - XorNot 14y ago
      Just look at the recent KDE git disaster. A lot of things went wrong there, but fundamentally the issue was ext4 silently returning bad data. The thing about fsck-like recovery tools is you need to have a failure mode in mind when you write them. ZFS can fix most of those types of errors thanks to the checksums and on-disk redundancy on the fly. Or at least tell you that something is now going wrong and which files are affected.

antonios 14y ago

ZFS needs no fsck. Period. http://www.c0t0d0s0.org/archives/6071-No,-ZFS-really-doesnt-need-a-fsck.html http://www.c0t0d0s0.org/archives/6071-No,-ZFS-really-doesnt-...

tjoff 14y ago
Actually. It does. Unless you assume that all hardware is perfect and that no bugs exists in ZFS (have been plenty of those in the past that just makes the whole FS completely worthless). Nut no, ZFS doesn't need fsck. Because, we don't want it to need one. Oh, and we don't want to spend the resources developing one. The only reason ZFS doesn't have a fsck tool is because in the enterprise world it doesn't need one. When it is needed you just restore from tape instead. It is that simple.
- reeenobo 14y ago
  Actually. It. Does. Not.