5 ms·

I feel like there has been kind of a weird concerted effort to push that zfs is bad due to this bug and how trust has been lost etcetera etcetera - super annoyi

I feel like there has been kind of a weird concerted effort to push that zfs is bad due to this bug and how trust has been lost etcetera etcetera - super annoying when most other filesystems just corrupt your data and nobody will ever know it happened. I’ve experienced bad data corruption on xfs, btrfs, ext2, and ext4. So far zfs is been nothing but perfect.

antongribok 3y ago
The reason, and the difference, is that all these other filesystems have check and repair (and sometimes multiple) tools. Please correct me, but ZFS has none.
- gavinhoward 3y ago
  You're completely wrong. ZFS's scrub is both a check and a repair tool. It's already saved some of my data.
  - hdjdkdbdbe 3y ago
    You are partly right. Zfs scrub is a repair tool when it has parity / mirrored copy of data to recreate it.
    - danparsonson 3y ago
      It's safe to make the assumption that the tool isn't magic.
    - Borealid 3y ago
      A scrub can also repair data when using `ncopies` greater than one even outside any mirrors or parity.
- deleted 3y ago
  [deleted]
- nightfly 3y ago
  Zfs has checksumming and scrubs, which can catch lots of data corruption that (most?) other filesystems can't catch catch at all
  - antongribok 3y ago
    I guess I meant it as having a separate repair tool, but yes, you're of course correct. To me it was always weird to not have a separate tool for being able to do an offline ZFS repair. With regards to data corruption, I mean, this is exactly why I moved to btrfs, because it's able to catch bit rot same as ZFS. I think ultimately this is why ZFS data corruption bugs always are such a big deal... It's because in a lot of tech circles ZFS is put on this infallible pedestal where it can never do any harm to your data.
    - AndrewDavis 3y ago
      Could you please explain what you mean by a separate repair tool kit?
      - lproven 3y ago
        I think he means an equivalent for `fsck`.
        db48x 3y ago
        `zfs scrub` is equivalent to `fsck`. It just has a different name, and it can fix more kinds of corruption (including corrupted data, which `fsck` for other file systems can never do), and can do so on line. Of course zfs can fix those same errors on line while doing ordinary reads, so really all `zfs scrub` does is read everything. `zfs scrub` is better than `fsck`.
        rincebrain 3y ago
        It is not. `zpool scrub` walks every block in the pool, and implicitly, you do checksums and other things while doing that. That's it. It's not doing any sort of logic bug repairs or cleanup or anything else. It's also not checking that you can, say, decrypt things, since that would mean you needed the keys to scrub.
- chungy 3y ago
  What good would ZFS be without that? The key differences are in two places: 1. Every administration task for ZFS is done online. You don't need to take your pool offline just to check for errors and repair them (if possible). (Mind that on FreeBSD, UFS can usually have a fsck done in the background while the file system is in-use. Just about zero Linux-native file systems have this capability) 2. "other filesytems" can only hope to detect and repair inconsistencies in their metadata structures. If your files are corrupted, they can't know and won't tell you that they are. ZFS checksums everything, including regular file data. It will repair regular file data too, if possible.
3np 3y ago
My experience has been aggravated by being simultaneously affected by https://github.com/openzfs/zfs/issues/11893 https://github.com/openzfs/zfs/issues/11893, which results in all zfs operations hanging (kill -9 is ineffective) until a power cycle. This round of upgrades was not fun. Debian bookworm users: You need to enable backports for zfs. Both bugs are fixed in backports but still present in stable. Let's do away with the tribalism in both ways? Calling it perfect is quite a stretch. I'm not saying "don't use zfs", just "you want to be prepared for the worst-case if your data matters". 3-2-1 and "validate your backups" still apply.
- buildbot 3y ago
  Perfect for me is not perfect for others of course. I have not run into the issue you linked, that sounds frustrating to deal with. Backups is the only (near) 100% way to insure you data I agree.
- sgarland 3y ago
  As someone currently rewriting Ansible plays to build Bookworm QEMU images with ZFS support, thank you for this incredibly niche and useful bit of information.
  - 3np 3y ago
    Np! Actually, in case you're able to share some of those plays in any form, that could be very useful here, however niche :)
    - sgarland 3y ago
      I have an older version [0] for Bullseye that you're welcome to take a look at. It's actually combined Ansible + Packer, so if you point it at a Proxmox host, you get an image in ~10-15 minutes. It's a fork of other work [1], modified and updated for Debian and Debian only. It also in absolutely no way has any best practices, which is why I'm doing a complete re-write with proper role dependencies, modularity, etc. When that's done, I'll be marking [0] as archived and linking to the new repo. The new version also replaces the incredibly hacky shell script builder (hooray dialog) with Python, and updates Packer's JSON with HCL2. [0]: https://github.com/stephanGarland/packer-proxmox-templates https://github.com/stephanGarland/packer-proxmox-templates [1]: https://github.com/chriswayg/packer-proxmox-templates https://github.com/chriswayg/packer-proxmox-templates
worthless-trash 3y ago
No, you're just hearing about ZFS corruption like you've heard corruption in other filesystems, this wont be the first and last. Its easy to feel like its being targeted with some kind of campaign, however the truth is rarely that exciting. The software is being used by more people, which means it will expose more bugs. No software is perfect, it just now starting to be abused enough to be important enough to be talked about. This stage is entirely normal.
- buildbot 3y ago
  But it's totally different than corruption in other filesystems. People are acting like upgrading to ZFS 2.2.0 ate all their data like XFS used to back in the bad old days. I remember once the power went out at my house, and the entire XFS filesystem was irreparably damaged. This bug is super hard to trigger, and has roots back to 2006 and only surfaced with a recent coreutils update. Yet there are numerous posts in the past few days verging on hysteria. I myself was very worried until I saw posts like the grand parent comment or the linked GitHub issues. I don't believe there's actually a campaign of ZFS haters sitting around in a storeroom somewhere posting to phoronix and HN, but it does feel like a lot of strongly held opinions are being unleashed now that ZFS had this bug. I'm pretty sure ZFS has been abused a lot - ten years ago I was helping test petabyte scale ZFS clusters, doing tons of concurrent IO and such. I'm pretty sure Sun and Oracle tested the crap out of it.
  - worthless-trash 3y ago
    > But it's totally different than corruption in other > filesystems. People are acting like upgrading to ZFS 2.2.0 > ate all their data like XFS used to back in the bad old > days. I remember once the power went out at my house, and > the entire XFS filesystem was irreparably damaged. Different bugs manifest differently due to the coding, this is just the same. ZFS wouldn't even have a hundredth of the testing of real-world use ext3/ext4 in hours. The large scale deployments of ext3/ext4 dwarf the usage of ZFS.
    - lproven 3y ago
moss2 3y ago
I'm unreasonably annoyed you used etcetera instead of et cetera. 'Et' is Latin for 'and'. The term means "and other things", not "andotherthings". No, I have nothing important to add to the ZFS discussion.
- frafra 3y ago
  etcetera is found in ~500 years old books, and et cetera itself comes from et caetera, from the Greek καὶ τὰ ἕτερα. Which one should we use then? Should we only use the Latin from 1 BCE/CE or the vulgar Latin that evolved after? :-) Maybe I do not feel it like a big issue, since many Latin languages use it a single word (such as Italian with eccetera, Spanish, Portuguese), due to the evolution of the language.
  - moss2 3y ago
    I guess even ~500 year old books can be wrong :-) I'm just joking. This is just a pet peeve of mine. Thank you for the fun lesson in etymology.
- buildbot 3y ago
  Did not know this, I don’t mind learning something. You’s think though that autocorrect would pick up on that…

antongribok 3y ago

The reason, and the difference, is that all these other filesystems have check and repair (and sometimes multiple) tools. Please correct me, but ZFS has none.

gavinhoward 3y ago
You're completely wrong. ZFS's scrub is both a check and a repair tool. It's already saved some of my data.
- hdjdkdbdbe 3y ago
  You are partly right. Zfs scrub is a repair tool when it has parity / mirrored copy of data to recreate it.
  - danparsonson 3y ago
    It's safe to make the assumption that the tool isn't magic.
  - Borealid 3y ago
    A scrub can also repair data when using `ncopies` greater than one even outside any mirrors or parity.
deleted 3y ago
[deleted]
nightfly 3y ago
Zfs has checksumming and scrubs, which can catch lots of data corruption that (most?) other filesystems can't catch catch at all
- antongribok 3y ago
  I guess I meant it as having a separate repair tool, but yes, you're of course correct. To me it was always weird to not have a separate tool for being able to do an offline ZFS repair. With regards to data corruption, I mean, this is exactly why I moved to btrfs, because it's able to catch bit rot same as ZFS. I think ultimately this is why ZFS data corruption bugs always are such a big deal... It's because in a lot of tech circles ZFS is put on this infallible pedestal where it can never do any harm to your data.
  - AndrewDavis 3y ago
    Could you please explain what you mean by a separate repair tool kit?
    - lproven 3y ago
      I think he means an equivalent for `fsck`.
      - db48x 3y ago
        `zfs scrub` is equivalent to `fsck`. It just has a different name, and it can fix more kinds of corruption (including corrupted data, which `fsck` for other file systems can never do), and can do so on line. Of course zfs can fix those same errors on line while doing ordinary reads, so really all `zfs scrub` does is read everything. `zfs scrub` is better than `fsck`.
        rincebrain 3y ago
        It is not. `zpool scrub` walks every block in the pool, and implicitly, you do checksums and other things while doing that. That's it. It's not doing any sort of logic bug repairs or cleanup or anything else. It's also not checking that you can, say, decrypt things, since that would mean you needed the keys to scrub.
chungy 3y ago
What good would ZFS be without that? The key differences are in two places: 1. Every administration task for ZFS is done online. You don't need to take your pool offline just to check for errors and repair them (if possible). (Mind that on FreeBSD, UFS can usually have a fsck done in the background while the file system is in-use. Just about zero Linux-native file systems have this capability) 2. "other filesytems" can only hope to detect and repair inconsistencies in their metadata structures. If your files are corrupted, they can't know and won't tell you that they are. ZFS checksums everything, including regular file data. It will repair regular file data too, if possible.

3np 3y ago

My experience has been aggravated by being simultaneously affected by https://github.com/openzfs/zfs/issues/11893 https://github.com/openzfs/zfs/issues/11893, which results in all zfs operations hanging (kill -9 is ineffective) until a power cycle. This round of upgrades was not fun. Debian bookworm users: You need to enable backports for zfs. Both bugs are fixed in backports but still present in stable. Let's do away with the tribalism in both ways? Calling it perfect is quite a stretch. I'm not saying "don't use zfs", just "you want to be prepared for the worst-case if your data matters". 3-2-1 and "validate your backups" still apply.

buildbot 3y ago
Perfect for me is not perfect for others of course. I have not run into the issue you linked, that sounds frustrating to deal with. Backups is the only (near) 100% way to insure you data I agree.
sgarland 3y ago
As someone currently rewriting Ansible plays to build Bookworm QEMU images with ZFS support, thank you for this incredibly niche and useful bit of information.
- 3np 3y ago
  Np! Actually, in case you're able to share some of those plays in any form, that could be very useful here, however niche :)
  - sgarland 3y ago
    I have an older version [0] for Bullseye that you're welcome to take a look at. It's actually combined Ansible + Packer, so if you point it at a Proxmox host, you get an image in ~10-15 minutes. It's a fork of other work [1], modified and updated for Debian and Debian only. It also in absolutely no way has any best practices, which is why I'm doing a complete re-write with proper role dependencies, modularity, etc. When that's done, I'll be marking [0] as archived and linking to the new repo. The new version also replaces the incredibly hacky shell script builder (hooray dialog) with Python, and updates Packer's JSON with HCL2. [0]: https://github.com/stephanGarland/packer-proxmox-templates https://github.com/stephanGarland/packer-proxmox-templates [1]: https://github.com/chriswayg/packer-proxmox-templates https://github.com/chriswayg/packer-proxmox-templates

worthless-trash 3y ago

No, you're just hearing about ZFS corruption like you've heard corruption in other filesystems, this wont be the first and last. Its easy to feel like its being targeted with some kind of campaign, however the truth is rarely that exciting. The software is being used by more people, which means it will expose more bugs. No software is perfect, it just now starting to be abused enough to be important enough to be talked about. This stage is entirely normal.

buildbot 3y ago
But it's totally different than corruption in other filesystems. People are acting like upgrading to ZFS 2.2.0 ate all their data like XFS used to back in the bad old days. I remember once the power went out at my house, and the entire XFS filesystem was irreparably damaged. This bug is super hard to trigger, and has roots back to 2006 and only surfaced with a recent coreutils update. Yet there are numerous posts in the past few days verging on hysteria. I myself was very worried until I saw posts like the grand parent comment or the linked GitHub issues. I don't believe there's actually a campaign of ZFS haters sitting around in a storeroom somewhere posting to phoronix and HN, but it does feel like a lot of strongly held opinions are being unleashed now that ZFS had this bug. I'm pretty sure ZFS has been abused a lot - ten years ago I was helping test petabyte scale ZFS clusters, doing tons of concurrent IO and such. I'm pretty sure Sun and Oracle tested the crap out of it.
- worthless-trash 3y ago
  > But it's totally different than corruption in other > filesystems. People are acting like upgrading to ZFS 2.2.0 > ate all their data like XFS used to back in the bad old > days. I remember once the power went out at my house, and > the entire XFS filesystem was irreparably damaged. Different bugs manifest differently due to the coding, this is just the same. ZFS wouldn't even have a hundredth of the testing of real-world use ext3/ext4 in hours. The large scale deployments of ext3/ext4 dwarf the usage of ZFS.
  - lproven 3y ago

moss2 3y ago

I'm unreasonably annoyed you used etcetera instead of et cetera. 'Et' is Latin for 'and'. The term means "and other things", not "andotherthings". No, I have nothing important to add to the ZFS discussion.

frafra 3y ago
etcetera is found in ~500 years old books, and et cetera itself comes from et caetera, from the Greek καὶ τὰ ἕτερα. Which one should we use then? Should we only use the Latin from 1 BCE/CE or the vulgar Latin that evolved after? :-) Maybe I do not feel it like a big issue, since many Latin languages use it a single word (such as Italian with eccetera, Spanish, Portuguese), due to the evolution of the language.
- moss2 3y ago
  I guess even ~500 year old books can be wrong :-) I'm just joking. This is just a pet peeve of mine. Thank you for the fun lesson in etymology.
buildbot 3y ago
Did not know this, I don’t mind learning something. You’s think though that autocorrect would pick up on that…