5 ms·

> You never see the same thing against someone running ZFS without regular scrubs or in RAIDZ1. ZFS doesn't have these kinds of hidden gotchas, and that's the

by chungy 5y ago

> You never see the same thing against someone running ZFS without regular scrubs or in RAIDZ1. ZFS doesn't have these kinds of hidden gotchas, and that's the key difference. Yeah ok somebody's being dumb if they never scrub and find out they have uncorrectable bad data come from two drives on a raidz1. That's exactly the advertised limitation of raidz1: it can survive a single complete drive failure, and can't repair data that has been corrupted on two (or more) drives at once. If you are in the scenario, as the GP was, that you have a two-disk mirror and regular scrubs have assured that one of the disks has only good data, and the other dies, ZFS won't corrupt the data on its own. If you try replacing the bad drive with another bad drive, eventually the bad drive will fail or produce so many errors that ZFS stops trying to use it, and you'll know. The pool will continue on with the good drive and tell you about it. Then you buy another replacement and hope that one is good. No surprises.

Arnavion 5y ago
ZFS is unfortunately not an option if you can't allow your kernel to be tainted, whether for idealistic reasons or for supportability reasons.
stryan 5y ago
>ZFS doesn't have these kinds of hidden gotchas, and that's the key difference. Yeah ok somebody's being dumb if they never scrub and find out they have uncorrectable bad data come from two drives on a raidz1. That's exactly the advertised limitation of raidz1: it can survive a single complete drive failure, and can't repair data that has been corrupted on two (or more) drives at once. Why is ZFS requiring scrubs and understanding the limitations of it's RAID implementations okay, but btrfs requiring scrubs and understanding the limitations of its RAID implementations "hidden gotchas"? > If you are in the scenario, as the GP was, that you have a two-disk mirror and regular scrubs have assured that one of the disks has only good data, and the other dies, ZFS won't corrupt the data on its own. Honestly, I don't know enough about GP's situation to really comment on what happened there. It could have been btrfs or perhaps they were using hardware RAID and the controller screwed up. ZFS is definitely very good in that regard and I want to be clear that I'm not saying ZFS is bad or that btrfs is better then it; I've been using ZFS much longer then I have btrfs, back before ZoL was a thing.
- chungy 5y ago
  > Why is ZFS requiring scrubs and understanding the limitations of it's RAID implementations okay, but btrfs requiring scrubs and understanding the limitations of its RAID implementations "hidden gotchas"? It read more like btrfs corrupting data despite good scrubbing practice; hosing the file system on the good drive instead of letting it remain good, for instance. If that's a misreading, that is where my position came from. Regular scrubs and understanding the limitations of redundancy models is good on both systems, yes. My own anecdotal evidence though: btrfs really does snag itself into surprising and disastrous situations at alarming frequency. Between being unable to reshape a pool (eg, removing a disk when plenty of free space exists) and not being safe with unclean shutdowns, it's hard to ever trust it. It even went a few good years where it seemed to be abandoned, but I guess since 2018 or so it's been picked up again.
  - stryan 5y ago
    Ah, I understand what you're saying now. Yeah, that's fair assuming it was btrfs's fault for the data loss. >Between being unable to reshape a pool (eg, removing a disk when plenty of free space exists) and not being safe with unclean shutdowns, it's hard to ever trust it. It even went a few good years where it seemed to be abandoned, but I guess since 2018 or so it's been picked up again. FYI, btrfs does support reshaping pool with the btrfs device commands.
    - chungy 5y ago
      > FYI, btrfs does support reshaping pool with the btrfs device commands. I've found it trivially easy to get btrfs stuck in a state where the commands to do so refuse to function.
vetinari 5y ago
Oh, don't worry, ZFS has a set of its own gotchas. My favorite is #8885.
- chungy 5y ago
  That issue sounds like a configuration error; I don't even encounter said "gotcha" on Linux systems. Even if you grant it to be a simple bug, it's not exactly a grave design issue with ZFS.
  - vetinari 5y ago
    It just kind of sucks, when it happens, you are thrown into recovery console at boot and there is no solution. From what I saw, it happened if the disk initialization took longer and ZoL looked for its pools before all disks were found. It hints at improper dependencies in ZoL startup.
- paulmd 5y ago
  for reference: this is referring to a systemd service script (systemd-udev-trigger/systemd-udev-settle) with a race condition where the pool may not be mounted by the time systemd tries to use it. https://github.com/openzfs/zfs/issues/8885 https://github.com/openzfs/zfs/issues/8885 that's (a) not really a bug in ZFS, and (b) "fails to boot sometimes" is pretty different from btrfs shitting the bed and corrupting its pool. There was one of those recently with ZFS iirc (and specifically only ZFS-on-Linux) but they are fairly rare and notable when they occur! (As a general statement, ZoL is less mature than ZFS-on-FreeBSD and likely (perhaps) to continue to be so given the licensing issues. I've also run into some problems where I can't send a dataset from FreeBSD to a ZoL pool (but rsync works fine). But again, generally bugs that actually lead to data loss are exceedingly rare.)
  - CorrectHorseBat 5y ago
    >As a general statement, ZoL is less mature than ZFS-on-FreeBSD That's strange since FreeBSD actually moved to ZoL a while ago

by chungy 5y ago

Arnavion 5y ago

ZFS is unfortunately not an option if you can't allow your kernel to be tainted, whether for idealistic reasons or for supportability reasons.

stryan 5y ago

>ZFS doesn't have these kinds of hidden gotchas, and that's the key difference. Yeah ok somebody's being dumb if they never scrub and find out they have uncorrectable bad data come from two drives on a raidz1. That's exactly the advertised limitation of raidz1: it can survive a single complete drive failure, and can't repair data that has been corrupted on two (or more) drives at once. Why is ZFS requiring scrubs and understanding the limitations of it's RAID implementations okay, but btrfs requiring scrubs and understanding the limitations of its RAID implementations "hidden gotchas"? > If you are in the scenario, as the GP was, that you have a two-disk mirror and regular scrubs have assured that one of the disks has only good data, and the other dies, ZFS won't corrupt the data on its own. Honestly, I don't know enough about GP's situation to really comment on what happened there. It could have been btrfs or perhaps they were using hardware RAID and the controller screwed up. ZFS is definitely very good in that regard and I want to be clear that I'm not saying ZFS is bad or that btrfs is better then it; I've been using ZFS much longer then I have btrfs, back before ZoL was a thing.

chungy 5y ago
> Why is ZFS requiring scrubs and understanding the limitations of it's RAID implementations okay, but btrfs requiring scrubs and understanding the limitations of its RAID implementations "hidden gotchas"? It read more like btrfs corrupting data despite good scrubbing practice; hosing the file system on the good drive instead of letting it remain good, for instance. If that's a misreading, that is where my position came from. Regular scrubs and understanding the limitations of redundancy models is good on both systems, yes. My own anecdotal evidence though: btrfs really does snag itself into surprising and disastrous situations at alarming frequency. Between being unable to reshape a pool (eg, removing a disk when plenty of free space exists) and not being safe with unclean shutdowns, it's hard to ever trust it. It even went a few good years where it seemed to be abandoned, but I guess since 2018 or so it's been picked up again.
- stryan 5y ago
  Ah, I understand what you're saying now. Yeah, that's fair assuming it was btrfs's fault for the data loss. >Between being unable to reshape a pool (eg, removing a disk when plenty of free space exists) and not being safe with unclean shutdowns, it's hard to ever trust it. It even went a few good years where it seemed to be abandoned, but I guess since 2018 or so it's been picked up again. FYI, btrfs does support reshaping pool with the btrfs device commands.
  - chungy 5y ago
    > FYI, btrfs does support reshaping pool with the btrfs device commands. I've found it trivially easy to get btrfs stuck in a state where the commands to do so refuse to function.

vetinari 5y ago

Oh, don't worry, ZFS has a set of its own gotchas. My favorite is #8885.

chungy 5y ago
That issue sounds like a configuration error; I don't even encounter said "gotcha" on Linux systems. Even if you grant it to be a simple bug, it's not exactly a grave design issue with ZFS.
- vetinari 5y ago
  It just kind of sucks, when it happens, you are thrown into recovery console at boot and there is no solution. From what I saw, it happened if the disk initialization took longer and ZoL looked for its pools before all disks were found. It hints at improper dependencies in ZoL startup.
paulmd 5y ago
for reference: this is referring to a systemd service script (systemd-udev-trigger/systemd-udev-settle) with a race condition where the pool may not be mounted by the time systemd tries to use it. https://github.com/openzfs/zfs/issues/8885 https://github.com/openzfs/zfs/issues/8885 that's (a) not really a bug in ZFS, and (b) "fails to boot sometimes" is pretty different from btrfs shitting the bed and corrupting its pool. There was one of those recently with ZFS iirc (and specifically only ZFS-on-Linux) but they are fairly rare and notable when they occur! (As a general statement, ZoL is less mature than ZFS-on-FreeBSD and likely (perhaps) to continue to be so given the licensing issues. I've also run into some problems where I can't send a dataset from FreeBSD to a ZoL pool (but rsync works fine). But again, generally bugs that actually lead to data loss are exceedingly rare.)
- CorrectHorseBat 5y ago
  >As a general statement, ZoL is less mature than ZFS-on-FreeBSD That's strange since FreeBSD actually moved to ZoL a while ago