4 ms·

The whole fsck discussion seems baffling to me. While I'm no ZFS expert, I've been using it for several years now and my understanding is this. Take what a nor

by deelowe 14y ago

The whole fsck discussion seems baffling to me. While I'm no ZFS expert, I've been using it for several years now and my understanding is this. Take what a normal fsck type tool does and build those features into the underlying FS and supporting toolchain. For what ZFS does and how it works, it really doesn't make sense to me at all for it to have an "fsck," whatever that means. Really, it's hard to even imagine what an "fsck" would do for zfs. You'd just end up rewriting bits of the toolchain or asking for the impossible. I asked this in the other thread, but I'll ask here again. Excluding semantics, what is it that people want fsck to do specifically that zfs doesn't provide a method for already? Seriously, the question to me seems akin to asking why manufacturers don't publish the rpm spec for SSDs. It's a really odd thing to ask and can't be answered without an exhaustive review of the mechanics of the system. I can't help but get the feeling that a lot of people complaining about ZFS have very little knowledge or familiarity with it and/or BSD/Unix in general. ZFS is not like any Linux FS. It doesn't use fstab, the toolchain is totally different, the FS is fundamentally different. It was built for Solaris and really reflects their ideology, which is completely foreign to people who only have familiarity with Linux. Accept it and move on or don't, but I've yet to see any evidence to back up these claims other than "this is what is done in Linux for everything else" which is just FUD.

tomku 14y ago
For most people, fsck and filesystems that rely on fsck are a known quantity. They understand that a filesystem does what it can to keep itself consistent, but that sometimes outside assistance is necessary in the form of fsck. When you show them ZFS and say "Oh, it doesn't need a fsck program", they assume that you're bullshitting them to cover up for the fact that it doesn't have a fsck program yet. As far as the Linux/Unix thing, I think you're reading way too far into it. Linux neither invented nor popularized filesystem consistency checking programs. Unix filesystems (such as UFS) often have consistency checking programs as well, though they might not be called "fsck". Windows has Scandisk. ZFS is the odd one out here, and it's not surprising for people to treat it as such. Give it time, and if ZFS's approach becomes more widespread, people will come around. tl;dr: It's about trust, not ignorance.
- deelowe 14y ago
  But ZFS has the ability to scrub pools, it can roll back transactions, and it can import of degraded arrays as well as many other things. Again, people seem to be failing to understand that the things fsck does are there either in the FS itself or in the tools. I stand by the argument that this whole debate is due to fear and a lack of understanding(aka ignorance). I brought up UFS as people who just use linux often similarly complain about the way BSD does "partitioning." The two debates seem very similar to me.
- pizza_ 14y ago
  Agreed. Which is why the correct answer is to create a ZFS fsck program that is a no-op to make these folks feel better.
bcantrill 14y ago
Yes, emphatically agreed. I co-founded Fishworks, the group within Sun that shipped a ZFS-based appliance. The product was (and, I add with mixed emotion, still is) very commercially successful, and we shipped hundreds of thousands of spindles running ZFS in production, enterprise environments. And today I'm at Joyent, where we run ZFS in production on tens of thousands of spindles and support software customers running many tens of thousands more. Across all of that experience -- which has (naturally) included plenty of pain -- I have never needed or wanted anything resembling a traditional "fsck" for ZFS. Those that decry ZFS's lack of a fsck simply don't understand (1) the semantics of ZFS and specifically of pool import, (2) the ability to rollback transactions and/or (3) the presence of zdb (which we've taken pains to document in illumos[1], the repository of record for ZFS). So please, take it from someone with a decade of production experience with ZFS: it does not need fsck. [1] http://illumos.org/man/1m/zdb http://illumos.org/man/1m/zdb
- notimetorelax 14y ago
  Reading about ZFS on Wikipedia and here [1] I see that there is a procedure called scrubbing. From [1]: "ZFS provides a mechanism to perform routine checking of all inconsistencies. This functionality, known as scrubbing, is commonly used in memory and other systems as a method of detecting and preventing errors before they result in hardware or software failure." So I guess "alias fsck='zpool scrub tank'" would quail everyone's concerns. [1] http://docs.oracle.com/cd/E19082-01/817-2271/gbbwa/ http://docs.oracle.com/cd/E19082-01/817-2271/gbbwa/
  - c0t0d0s0 14y ago
    Scrubbing is more like reading the block, calculate the checksum. It's the same as stored for the block (not at the block, important difference)? Nice. Not the same checksum. Go to the mirror and fetch the correct one, respectively use an algorithm to calculate the missing block from the raidz1/2/3 set and write it to the disk with the error.
    - notimetorelax 14y ago
cyanoacry 14y ago
I've had to deal with one scenario where ZFS fsck would have been very useful, though it comes up rarely: corruption leading to valid checksums, but bad ZFS data. In my case[1], our setup died due to some power problems, and somehow, NULL pointers got written to disk with valid checksums. Normally this wouldn't happen, but when it does, it's a PITA to debug because trying to traverse/read the disk gives a kernel fault, instead of a segfault as you might see in a user-level fsck program. This was a real pain, as we had encrypted disks, and every reboot meant going through the disk attach steps (enter password, etc etc), every time. As a result, a userspace implementation of scrubbing would be useful since in this sort of rare instance, I'd be able to probe the fsck process with a good debugger and see why it's crashing. Since it's in userspace, the fsck program can also quit more sanely, with a full report on where it found the corruption. I was able to get my data back via some ad-hoc patches, but it was an... interesting experience having to debug in kernel vs in userspace. zdb isn't a substitute for most of these things as the on-disk compression and RAIDZ sharding makes it difficult to actually see the raw data structures. Max Bruning wrote a post a while back with an on-disk data walkthrough[2], where he wrote some patches to fix this, but they haven't made their way upstream yet. Additionally, FreeBSD and Linux don't have mdb. :( [1] http://lists.freebsd.org/pipermail/freebsd-fs/2012-November/015808.html http://lists.freebsd.org/pipermail/freebsd-fs/2012-November/... [2] http://mbruning.blogspot.com/2009/12/zfs-raidz-data-walk.html http://mbruning.blogspot.com/2009/12/zfs-raidz-data-walk.htm... edit: formatting
- mrbruning 14y ago
  I wrote that code several years ago. It is not quite the right way to go. zdb now does decompression, though slightly differently from what I implemented. Syntax is: zdb -R poolname vdev:offset:size:d The "d" at the end says to decompress. zdb tries different decompression algorithms until it finds one that is correct. As for my mdb changes, I really think mdb should be able to pick up kernel ctf info so that it can print data structures on disk. That I could probably get working on illumos fairly easily. My method used zdb to get the data uncompressed, then used mdb to print it out ala ::print. I actually think something like "offset::zprint [decompression] type" in mdb is the way to go. It would mean no need for zdb, which usually gives too much or not enough, and is not interactive (hence, not really a good debugger as far as I'm concerned). Better would be: # mdb -z poolname 20000::walk uberblock | ::print -t uberblock_t And from there, something like: offset::zprint lzjb objset_phys_t where offset comes from a DVA displayed in the uberblock_t. Some people seem to get my idea and think it's good. Others either don't get it, or don't care. Someone like Delphix might really like it. Just my 2 cents. max

by deelowe 14y ago

tomku 14y ago

For most people, fsck and filesystems that rely on fsck are a known quantity. They understand that a filesystem does what it can to keep itself consistent, but that sometimes outside assistance is necessary in the form of fsck. When you show them ZFS and say "Oh, it doesn't need a fsck program", they assume that you're bullshitting them to cover up for the fact that it doesn't have a fsck program yet. As far as the Linux/Unix thing, I think you're reading way too far into it. Linux neither invented nor popularized filesystem consistency checking programs. Unix filesystems (such as UFS) often have consistency checking programs as well, though they might not be called "fsck". Windows has Scandisk. ZFS is the odd one out here, and it's not surprising for people to treat it as such. Give it time, and if ZFS's approach becomes more widespread, people will come around. tl;dr: It's about trust, not ignorance.

deelowe 14y ago
But ZFS has the ability to scrub pools, it can roll back transactions, and it can import of degraded arrays as well as many other things. Again, people seem to be failing to understand that the things fsck does are there either in the FS itself or in the tools. I stand by the argument that this whole debate is due to fear and a lack of understanding(aka ignorance). I brought up UFS as people who just use linux often similarly complain about the way BSD does "partitioning." The two debates seem very similar to me.
pizza_ 14y ago
Agreed. Which is why the correct answer is to create a ZFS fsck program that is a no-op to make these folks feel better.

bcantrill 14y ago

Yes, emphatically agreed. I co-founded Fishworks, the group within Sun that shipped a ZFS-based appliance. The product was (and, I add with mixed emotion, still is) very commercially successful, and we shipped hundreds of thousands of spindles running ZFS in production, enterprise environments. And today I'm at Joyent, where we run ZFS in production on tens of thousands of spindles and support software customers running many tens of thousands more. Across all of that experience -- which has (naturally) included plenty of pain -- I have never needed or wanted anything resembling a traditional "fsck" for ZFS. Those that decry ZFS's lack of a fsck simply don't understand (1) the semantics of ZFS and specifically of pool import, (2) the ability to rollback transactions and/or (3) the presence of zdb (which we've taken pains to document in illumos[1], the repository of record for ZFS). So please, take it from someone with a decade of production experience with ZFS: it does not need fsck. [1] http://illumos.org/man/1m/zdb http://illumos.org/man/1m/zdb

notimetorelax 14y ago
Reading about ZFS on Wikipedia and here [1] I see that there is a procedure called scrubbing. From [1]: "ZFS provides a mechanism to perform routine checking of all inconsistencies. This functionality, known as scrubbing, is commonly used in memory and other systems as a method of detecting and preventing errors before they result in hardware or software failure." So I guess "alias fsck='zpool scrub tank'" would quail everyone's concerns. [1] http://docs.oracle.com/cd/E19082-01/817-2271/gbbwa/ http://docs.oracle.com/cd/E19082-01/817-2271/gbbwa/
- c0t0d0s0 14y ago
  Scrubbing is more like reading the block, calculate the checksum. It's the same as stored for the block (not at the block, important difference)? Nice. Not the same checksum. Go to the mirror and fetch the correct one, respectively use an algorithm to calculate the missing block from the raidz1/2/3 set and write it to the disk with the error.
  - notimetorelax 14y ago

cyanoacry 14y ago

I've had to deal with one scenario where ZFS fsck would have been very useful, though it comes up rarely: corruption leading to valid checksums, but bad ZFS data. In my case[1], our setup died due to some power problems, and somehow, NULL pointers got written to disk with valid checksums. Normally this wouldn't happen, but when it does, it's a PITA to debug because trying to traverse/read the disk gives a kernel fault, instead of a segfault as you might see in a user-level fsck program. This was a real pain, as we had encrypted disks, and every reboot meant going through the disk attach steps (enter password, etc etc), every time. As a result, a userspace implementation of scrubbing would be useful since in this sort of rare instance, I'd be able to probe the fsck process with a good debugger and see why it's crashing. Since it's in userspace, the fsck program can also quit more sanely, with a full report on where it found the corruption. I was able to get my data back via some ad-hoc patches, but it was an... interesting experience having to debug in kernel vs in userspace. zdb isn't a substitute for most of these things as the on-disk compression and RAIDZ sharding makes it difficult to actually see the raw data structures. Max Bruning wrote a post a while back with an on-disk data walkthrough[2], where he wrote some patches to fix this, but they haven't made their way upstream yet. Additionally, FreeBSD and Linux don't have mdb. :( [1] http://lists.freebsd.org/pipermail/freebsd-fs/2012-November/015808.html http://lists.freebsd.org/pipermail/freebsd-fs/2012-November/... [2] http://mbruning.blogspot.com/2009/12/zfs-raidz-data-walk.html http://mbruning.blogspot.com/2009/12/zfs-raidz-data-walk.htm... edit: formatting

mrbruning 14y ago
I wrote that code several years ago. It is not quite the right way to go. zdb now does decompression, though slightly differently from what I implemented. Syntax is: zdb -R poolname vdev:offset:size:d The "d" at the end says to decompress. zdb tries different decompression algorithms until it finds one that is correct. As for my mdb changes, I really think mdb should be able to pick up kernel ctf info so that it can print data structures on disk. That I could probably get working on illumos fairly easily. My method used zdb to get the data uncompressed, then used mdb to print it out ala ::print. I actually think something like "offset::zprint [decompression] type" in mdb is the way to go. It would mean no need for zdb, which usually gives too much or not enough, and is not interactive (hence, not really a good debugger as far as I'm concerned). Better would be: # mdb -z poolname 20000::walk uberblock | ::print -t uberblock_t And from there, something like: offset::zprint lzjb objset_phys_t where offset comes from a DVA displayed in the uberblock_t. Some people seem to get my idea and think it's good. Others either don't get it, or don't care. Someone like Delphix might really like it. Just my 2 cents. max