4 ms·

>but for residential usage, it's totally reasonable to accept the risk. Polite disagree. Data integrity is the natural expectation humans have from computers,

by snvzz 2y ago

>but for residential usage, it's totally reasonable to accept the risk. Polite disagree. Data integrity is the natural expectation humans have from computers, and thus we should stick to filesystems with data checksums such as ZFS, as well as ECC memory.

rollcat 2y ago
> we should stick to filesystems with data checksums such as ZFS, as well as ECC memory. While I don't disagree with this statement, consider the reality: - APFS has metadata checksums, but no data checksums. WTF Apple? - Very few Linux distributions ship zfs.ko (&spl.ko); those that do, theoretically face a legal risk (any kernel contributor could sue them for breaching the GPL); rebuilding the driver from source is awkward (even with e.g. DKMS), pulls more power, takes time, and may randomly leave your system unbootable (YES it happened to me once). - Linux itself explicitly treats ZFS as unsupported; loading the module taints the kernel. - FreeBSD is great, and is actually making great progress catching up with Linux on the desktop. Still, it is a catch-up game. I also don't want to install a system that needs to install another guest system to actually run the programs I need. - There are no practical alternatives to ZFS that even come close; sibling comment complains about btrfs data loss. I never had the guts to try btrfs in production after all the horror stories I've heard over the decade+. - ECC memory on laptops is practically unheard of, save for a couple niche Thinkpad models; and comes with large premiums on desktops. What are the practical choices for people who do not want to cosplay as sysadmins?
- louwrentius 2y ago
  I feel that on HN people tend to be a bit pedantic about topics like data integrity, and in business settings I actually agree with them. But for residential use, risks are just different and as you point out, you have no options except to only use a desktop workstation with ECC. People like/need laptops so that’s not realistic for most people. Just run Linux/freebsd with ZFS isn’t reasonable advice to me. What I feel most strongly about is that it’s all about circumstances, context and risk evaluation. And I see so much blanket absolutist statements that don’t think about the reality of life and people’s circumstances.
- nubinetwork 2y ago
  > - Very few Linux distributions ship zfs.ko (&spl.ko); those that do, theoretically face a legal risk (any kernel contributor could sue them for breaching the GPL) > - Linux itself explicitly treats ZFS as unsupported; loading the module taints the kernel. So modify the ZFS source so it appears as a external GPL module... just don't tell anyone or distribute it... I can't say much about dracut or having to build the module from source... as a Gentoo user, I do it about once a month without any issues...
  - rollcat 2y ago
    > So modify the ZFS source Way to miss the point
    - nubinetwork 2y ago
      Not really... the complaint was over licensing and tainting the kernel... so just tell the kernel it's not a CDDL module... problem solved.
      - rollcat 2y ago
        > What are the practical choices for people who do not want to cosplay as sysadmins? The specific complaint is not at all about the kernel identifying itself as tainted, the specific complaint is about the kernel developers' unyielding unwillingness to support any scenario where ZFS is concerned, thus leaving one with even more "sysadmin duties". I want to use my computer, not serve it.
- defrost 2y ago
  Off the shelf commodity home NAS systems with ZFS onboard? eg: https://www.qnap.com/en-au/operating-system/quts-hero https://www.qnap.com/en-au/operating-system/quts-hero IIRC (been a while since I messed with QNAP) QuTS hero would be a modded Debian install with ZFS baked in and a web based admin dasboard. https://old.reddit.com/r/qnap/comments/15b9a0u/qts_or_quts_hero_how_to_choose/ https://old.reddit.com/r/qnap/comments/15b9a0u/qts_or_quts_h... As a rule of thumb (IMHO) steer clear of commodity NAS Cloud add ons, such things attract ransomware hackers like flies to a tip whether it's QNAP, Synology, or InsertVendorHere.
- freeone3000 2y ago
  “Tainting” the kernel doesn’t affect operations, though. You’re not allowed to redistribute it with changes — but you, as an entity, can freely use ZFS and the kernel together without restriction. Linux plus zfs works fine.
- simoncion 2y ago
  > I never had the guts to try btrfs in production after all the horror stories I've heard over the decade+. I've been running btrfs as the primary filesystem for all of my desktop machines since shortly after the on-disk format stabilized and the extX->btrfs in-place converter appeared [0], and for my home servers for the past ~five years. In the first few years after I started using it on my desktop machines, I had four or five "btrfs shit the bed and trashed some of my data" incidents. I've had zero issues in the past ~ten years. At $DAYJOB we use btrfs as the filesystem for our CI workers and have been doing so for years. Its snapshot functionality makes creating the containers for CI jobs instantaneous, and we've had zero problems with it. I can think of a few things that might separate me from the folks who report issues that they've had within the past five-or-ten years: * I don't use ANY of the built-in btrfs RAID stuff. * I deploy btrfs ON TOP of LVM2 LVs, rather than using its built-in volume management stuff. [1] * I WAS going to say "I use ECC RAM", but one of my desktop machines does not and can never have ECC RAM, so this isn't likely a factor. The BTRFS features I use at home are snapshotting (for coherent point-in-time backups), transparent compression, the built-in CoW features, and the built-in checksumming features. At work, we use all of those except for compression, and don't use snapshots for backup but for container volume cloning. [0] If memory serves, this was around the time when the OCZ Vertex LE was hot, hot shit. [1] This has actually turned out to be a really cool decision, as it has permitted me to do low- or no- downtime disk replacement or repartitioning by moving live data off of local PVs and on to PVs attached via USB or via NBD.
  - curt15 2y ago
    >At $DAYJOB we use btrfs as the filesystem for our CI workers and have been doing so for years. Its snapshot functionality makes creating the containers for CI jobs instantaneous, and we've had zero problems with it. $DAYJOB == "facebook"?
    - simoncion 2y ago
      > $DAYJOB == "facebook"? Nah. AFAIK, we don't have any Linux kernel gurus on the payroll... we're ordinary users just like most everyone else.
deleted 2y ago
[deleted]
naming_the_user 2y ago
I agree. I think that the author may not have experienced these sorts of errors before. Yes, the average person may not care about experiencing a couple of bit flips per year and losing the odd pixel or block of a JPEG, but they will care if some cable somewhere or transfer or bad RAM chip or whatever else manages to destroy a significant amount of data before they notice it.
- AndrewDavis 2y ago
  I had a significant data loss years ago. I was young and only had a deskop, so all my data was there. So I purchased a 300GB external usb drive to use for periodic backup. It was all manual copy/paste files across with no real schedule, but it was fine for the time and life was good. Over time my data grew and the 300GB drive wasn't large enough to store it all. For a while some of it wasnt backed up (I was young with much less disposable income). Eventually I purchased a 500GB drive. But what I didn't know is my desktop drive was dying. Bits were flipping, a lot of them. So when I did my first backup with the new drive I copied all my data off my desktop along with the corruption. It was months before I realised a huge amount of my files were corrupted. By that point I'd wiped the old backup drive to give to my Mum to do her own backups. My data was long gone. Once I discovered ZFS I jumped on it. It was the exact thing that would have prevented this because I could have detected the corruption when I purchased the new backup drive and did the initial backup to it. (I made up the drive sizes because I can't remember, but the ratios will be about right).
  - 369548684892826 2y ago
    There’s something disturbing about the idea of silent data loss, it totally undermines the peace of mind of having backups. ZFS is good, but you can also just run rsync periodically with checksum and dryrun args and check the output for diffs.
    - AndrewDavis 2y ago
      Absolutely, if you can't use a filesystem with checksums (zfs, btrfs, bcachefs) then rsync is a great idea. I think filesystem checksums have one big advantage vs rsync. With rsync if there's a difference it isn't clear which one is wrong.
mustache_kimono 2y ago
> Data integrity is the natural expectation humans have from computers I've said it once, and I'll say it again: the only reason ZFS isn't the norm is because we all once lived through a primordial era when it didn't exist. No serious person designing a filesystem today would say it's okay to misplace your data. Not long ago, on this forum, someone told me that ZFS is only good because it had no competitors in its space. Which is kind of like saying the heavyweight champ is only good because no one else could compete.
- hulitu 2y ago
  > the only reason ZFS isn't the norm is because we all once lived through a primordial era when it didn't exist. There were good filesystems before ZFS. I would love to have a versioning filesystem like Apollo had.
- wolrah 2y ago
  To paraphrase, "ZFS is the worst filesystem, except for all those other filesystems that have been tried from time to time." It's far from perfect, but it has no peers. I spent many years stubbornly using btrfs and lost data multiple times. Never once did the redundancy I had supposedly configured actually do anything to help me. ZFS has identified corruption caused by bad memory and a bad CPU and let me know immediately which files were damaged.
- Too 2y ago
  The reason ZFS isn't the norm is because it historically was difficult to set up. Outside of NAS solutions, it's only since Ubuntu 20.04 it has been supported out of the box on any high profile customer facing OS. The reliability of the early versions was also questionable, with high zsys cpu usage and some times arcane commands needed to rebuild pools. Anecdotally, I've had to support lots of friends with zfs issues, never so with other file systems. The data always comes back, it's just that it needs petting. Earlier, there used to be lot of fears around the license, with Torvalds advising against its use, both for that reason and for lack of maintainers. Now i believe that has been mostly ironed out and should be less of an issue.
  - 2y ago
Timshel 2y ago
Just checked my scrub history, for 20TB on consumer hardware during the last two years it repaired twice around 2 and 4 blocks each time. So not much but at the same time with a special kind of luck might have been on an encrypted archive ^^.
- nolok 2y ago
  That's all fine and good until that one random lone broken block stops you from opening that file you really need.
  - lazide 2y ago
    Or in my case, a key filesystem metadata block that ruins everything. :s
    - pbhjpbhj 2y ago
      I only know about FAT but these "key file metadata blocks" are redundant, so you need really special double-plus bad luck to do that.
      - Szpadel 2y ago
        so I can consider myself very lucky and unlucky at the same time. I had data corruption on zfs filesystem that destroyed whole pool to unrecoverable state (zfs was segfaulting while trying to import, all recovery zfs features where crashing zfs module and required reboot) the lucky part is that this happened just after (something like next day) I migrated whole pool to another (bigger) server/pool so that system was already scheduled for full disk wipe
        howard941 2y ago
        This happened to me too. The root cause was a bad memory stick.
      - lazide 2y ago
louwrentius 2y ago
Most people don’t run ZFS on their laptop, desktop so let’s not pretend it’s such a huge deal.
- BSDobelix 2y ago
  Most people run Windows on their Laptop (without ReFS), and many people use paid data restore services if something "important" gets missing/corrupt. >let’s not pretend it’s such a huge deal Depends on the importance of you data right?
  - louwrentius 2y ago
    I bet even 99.9% of HN visitors don’t run ZFS on their laptop/desktop. Basically we all take this risk except for a few dedicated nerds. Everything has a price and people like to have their uncorrupted files, but not at all cost.
    - rabf 2y ago
      I find this thinking difficult to reconcile. When I setup up my workstation it does usually take me half a day to sort out an encrypted rootfs mirrored volume with zfsbootmenu + linux, but after that its all set for the next decade. A small price for the peace of mind it affords.
    - BSDobelix 2y ago
      >Everything has a price and people like to have their uncorrupted files, but not at all cost. I have ZFS on nearly all machines and i don't have any cost, the opposite is actually true because of compression, snapshots an boot environment. >I bet even 99.9% of HN visitors don’t run ZFS on their laptop/desktop I really don't care that much what others run since 99.9% of HN don't run FreeBSD...probably, well except HN itself ;)
mytailorisrich 2y ago
I think this is over the top for standard residential usage. Make sure you have good continuous backups and perhaps RAID 1 on your file server (if you have one) to save efforts in case a disk fails and you are more than covered.

by snvzz 2y ago

rollcat 2y ago

> we should stick to filesystems with data checksums such as ZFS, as well as ECC memory. While I don't disagree with this statement, consider the reality: - APFS has metadata checksums, but no data checksums. WTF Apple? - Very few Linux distributions ship zfs.ko (&spl.ko); those that do, theoretically face a legal risk (any kernel contributor could sue them for breaching the GPL); rebuilding the driver from source is awkward (even with e.g. DKMS), pulls more power, takes time, and may randomly leave your system unbootable (YES it happened to me once). - Linux itself explicitly treats ZFS as unsupported; loading the module taints the kernel. - FreeBSD is great, and is actually making great progress catching up with Linux on the desktop. Still, it is a catch-up game. I also don't want to install a system that needs to install another guest system to actually run the programs I need. - There are no practical alternatives to ZFS that even come close; sibling comment complains about btrfs data loss. I never had the guts to try btrfs in production after all the horror stories I've heard over the decade+. - ECC memory on laptops is practically unheard of, save for a couple niche Thinkpad models; and comes with large premiums on desktops. What are the practical choices for people who do not want to cosplay as sysadmins?

louwrentius 2y ago
I feel that on HN people tend to be a bit pedantic about topics like data integrity, and in business settings I actually agree with them. But for residential use, risks are just different and as you point out, you have no options except to only use a desktop workstation with ECC. People like/need laptops so that’s not realistic for most people. Just run Linux/freebsd with ZFS isn’t reasonable advice to me. What I feel most strongly about is that it’s all about circumstances, context and risk evaluation. And I see so much blanket absolutist statements that don’t think about the reality of life and people’s circumstances.
nubinetwork 2y ago
> - Very few Linux distributions ship zfs.ko (&spl.ko); those that do, theoretically face a legal risk (any kernel contributor could sue them for breaching the GPL) > - Linux itself explicitly treats ZFS as unsupported; loading the module taints the kernel. So modify the ZFS source so it appears as a external GPL module... just don't tell anyone or distribute it... I can't say much about dracut or having to build the module from source... as a Gentoo user, I do it about once a month without any issues...
- rollcat 2y ago
  > So modify the ZFS source Way to miss the point
  - nubinetwork 2y ago
    Not really... the complaint was over licensing and tainting the kernel... so just tell the kernel it's not a CDDL module... problem solved.
    - rollcat 2y ago
      > What are the practical choices for people who do not want to cosplay as sysadmins? The specific complaint is not at all about the kernel identifying itself as tainted, the specific complaint is about the kernel developers' unyielding unwillingness to support any scenario where ZFS is concerned, thus leaving one with even more "sysadmin duties". I want to use my computer, not serve it.
defrost 2y ago
Off the shelf commodity home NAS systems with ZFS onboard? eg: https://www.qnap.com/en-au/operating-system/quts-hero https://www.qnap.com/en-au/operating-system/quts-hero IIRC (been a while since I messed with QNAP) QuTS hero would be a modded Debian install with ZFS baked in and a web based admin dasboard. https://old.reddit.com/r/qnap/comments/15b9a0u/qts_or_quts_hero_how_to_choose/ https://old.reddit.com/r/qnap/comments/15b9a0u/qts_or_quts_h... As a rule of thumb (IMHO) steer clear of commodity NAS Cloud add ons, such things attract ransomware hackers like flies to a tip whether it's QNAP, Synology, or InsertVendorHere.
freeone3000 2y ago
“Tainting” the kernel doesn’t affect operations, though. You’re not allowed to redistribute it with changes — but you, as an entity, can freely use ZFS and the kernel together without restriction. Linux plus zfs works fine.
simoncion 2y ago
> I never had the guts to try btrfs in production after all the horror stories I've heard over the decade+. I've been running btrfs as the primary filesystem for all of my desktop machines since shortly after the on-disk format stabilized and the extX->btrfs in-place converter appeared [0], and for my home servers for the past ~five years. In the first few years after I started using it on my desktop machines, I had four or five "btrfs shit the bed and trashed some of my data" incidents. I've had zero issues in the past ~ten years. At $DAYJOB we use btrfs as the filesystem for our CI workers and have been doing so for years. Its snapshot functionality makes creating the containers for CI jobs instantaneous, and we've had zero problems with it. I can think of a few things that might separate me from the folks who report issues that they've had within the past five-or-ten years: * I don't use ANY of the built-in btrfs RAID stuff. * I deploy btrfs ON TOP of LVM2 LVs, rather than using its built-in volume management stuff. [1] * I WAS going to say "I use ECC RAM", but one of my desktop machines does not and can never have ECC RAM, so this isn't likely a factor. The BTRFS features I use at home are snapshotting (for coherent point-in-time backups), transparent compression, the built-in CoW features, and the built-in checksumming features. At work, we use all of those except for compression, and don't use snapshots for backup but for container volume cloning. [0] If memory serves, this was around the time when the OCZ Vertex LE was hot, hot shit. [1] This has actually turned out to be a really cool decision, as it has permitted me to do low- or no- downtime disk replacement or repartitioning by moving live data off of local PVs and on to PVs attached via USB or via NBD.
- curt15 2y ago
  >At $DAYJOB we use btrfs as the filesystem for our CI workers and have been doing so for years. Its snapshot functionality makes creating the containers for CI jobs instantaneous, and we've had zero problems with it. $DAYJOB == "facebook"?
  - simoncion 2y ago
    > $DAYJOB == "facebook"? Nah. AFAIK, we don't have any Linux kernel gurus on the payroll... we're ordinary users just like most everyone else.

deleted 2y ago

[deleted]

naming_the_user 2y ago

I agree. I think that the author may not have experienced these sorts of errors before. Yes, the average person may not care about experiencing a couple of bit flips per year and losing the odd pixel or block of a JPEG, but they will care if some cable somewhere or transfer or bad RAM chip or whatever else manages to destroy a significant amount of data before they notice it.

AndrewDavis 2y ago
I had a significant data loss years ago. I was young and only had a deskop, so all my data was there. So I purchased a 300GB external usb drive to use for periodic backup. It was all manual copy/paste files across with no real schedule, but it was fine for the time and life was good. Over time my data grew and the 300GB drive wasn't large enough to store it all. For a while some of it wasnt backed up (I was young with much less disposable income). Eventually I purchased a 500GB drive. But what I didn't know is my desktop drive was dying. Bits were flipping, a lot of them. So when I did my first backup with the new drive I copied all my data off my desktop along with the corruption. It was months before I realised a huge amount of my files were corrupted. By that point I'd wiped the old backup drive to give to my Mum to do her own backups. My data was long gone. Once I discovered ZFS I jumped on it. It was the exact thing that would have prevented this because I could have detected the corruption when I purchased the new backup drive and did the initial backup to it. (I made up the drive sizes because I can't remember, but the ratios will be about right).
- 369548684892826 2y ago
  There’s something disturbing about the idea of silent data loss, it totally undermines the peace of mind of having backups. ZFS is good, but you can also just run rsync periodically with checksum and dryrun args and check the output for diffs.
  - AndrewDavis 2y ago
    Absolutely, if you can't use a filesystem with checksums (zfs, btrfs, bcachefs) then rsync is a great idea. I think filesystem checksums have one big advantage vs rsync. With rsync if there's a difference it isn't clear which one is wrong.

mustache_kimono 2y ago

> Data integrity is the natural expectation humans have from computers I've said it once, and I'll say it again: the only reason ZFS isn't the norm is because we all once lived through a primordial era when it didn't exist. No serious person designing a filesystem today would say it's okay to misplace your data. Not long ago, on this forum, someone told me that ZFS is only good because it had no competitors in its space. Which is kind of like saying the heavyweight champ is only good because no one else could compete.

hulitu 2y ago
> the only reason ZFS isn't the norm is because we all once lived through a primordial era when it didn't exist. There were good filesystems before ZFS. I would love to have a versioning filesystem like Apollo had.
wolrah 2y ago
To paraphrase, "ZFS is the worst filesystem, except for all those other filesystems that have been tried from time to time." It's far from perfect, but it has no peers. I spent many years stubbornly using btrfs and lost data multiple times. Never once did the redundancy I had supposedly configured actually do anything to help me. ZFS has identified corruption caused by bad memory and a bad CPU and let me know immediately which files were damaged.
Too 2y ago
The reason ZFS isn't the norm is because it historically was difficult to set up. Outside of NAS solutions, it's only since Ubuntu 20.04 it has been supported out of the box on any high profile customer facing OS. The reliability of the early versions was also questionable, with high zsys cpu usage and some times arcane commands needed to rebuild pools. Anecdotally, I've had to support lots of friends with zfs issues, never so with other file systems. The data always comes back, it's just that it needs petting. Earlier, there used to be lot of fears around the license, with Torvalds advising against its use, both for that reason and for lack of maintainers. Now i believe that has been mostly ironed out and should be less of an issue.
- 2y ago

Timshel 2y ago

Just checked my scrub history, for 20TB on consumer hardware during the last two years it repaired twice around 2 and 4 blocks each time. So not much but at the same time with a special kind of luck might have been on an encrypted archive ^^.

nolok 2y ago
That's all fine and good until that one random lone broken block stops you from opening that file you really need.
- lazide 2y ago
  Or in my case, a key filesystem metadata block that ruins everything. :s
  - pbhjpbhj 2y ago
    I only know about FAT but these "key file metadata blocks" are redundant, so you need really special double-plus bad luck to do that.
    - Szpadel 2y ago
      so I can consider myself very lucky and unlucky at the same time. I had data corruption on zfs filesystem that destroyed whole pool to unrecoverable state (zfs was segfaulting while trying to import, all recovery zfs features where crashing zfs module and required reboot) the lucky part is that this happened just after (something like next day) I migrated whole pool to another (bigger) server/pool so that system was already scheduled for full disk wipe
      - howard941 2y ago
        This happened to me too. The root cause was a bad memory stick.
    - lazide 2y ago

louwrentius 2y ago

Most people don’t run ZFS on their laptop, desktop so let’s not pretend it’s such a huge deal.

BSDobelix 2y ago
Most people run Windows on their Laptop (without ReFS), and many people use paid data restore services if something "important" gets missing/corrupt. >let’s not pretend it’s such a huge deal Depends on the importance of you data right?
- louwrentius 2y ago
  I bet even 99.9% of HN visitors don’t run ZFS on their laptop/desktop. Basically we all take this risk except for a few dedicated nerds. Everything has a price and people like to have their uncorrupted files, but not at all cost.
  - rabf 2y ago
    I find this thinking difficult to reconcile. When I setup up my workstation it does usually take me half a day to sort out an encrypted rootfs mirrored volume with zfsbootmenu + linux, but after that its all set for the next decade. A small price for the peace of mind it affords.
  - BSDobelix 2y ago
    >Everything has a price and people like to have their uncorrupted files, but not at all cost. I have ZFS on nearly all machines and i don't have any cost, the opposite is actually true because of compression, snapshots an boot environment. >I bet even 99.9% of HN visitors don’t run ZFS on their laptop/desktop I really don't care that much what others run since 99.9% of HN don't run FreeBSD...probably, well except HN itself ;)

mytailorisrich 2y ago

I think this is over the top for standard residential usage. Make sure you have good continuous backups and perhaps RAID 1 on your file server (if you have one) to save efforts in case a disk fails and you are more than covered.