8 ms·

I've been using it at home via FreeNAS after losing a bunch of old files to bitrot.

by zf00002 11y ago

StavrosK 11y ago
How would ZFS help there? If you don't scrub, you lose files whatever fs you use.
- abrookewood 11y ago
  He never said he didn't scrub ... assuming he did (and he held 2+ copies), then the files are restored. Even if he only had a single copy, ZFS would at least tell you what files are corrupt, allowing you to restore them from backup.
  - StavrosK 11y ago
    I guess that's true, I was thinking about the case of the redundant array (otherwise the files wouldn't be "lost").
    - dmm 11y ago
      Bitrot still affects traditional RAID. > As much as it affects ZFS, that's my point. That's not true. RAID protects you against disk failure but bitrot is caused by unrecoverable read failures. Check out this ars article: http://arstechnica.com/information-technology/2014/01/bitrot-and-atomic-cows-inside-next-gen-filesystems/ http://arstechnica.com/information-technology/2014/01/bitrot...
      - StavrosK 11y ago
        As much as it affects ZFS, that's my point.
      - ryao 11y ago
        RAID can handle unrecoverable read failures. What it cannot handle is silent corruption where bad data is returned as if it were the correct data.
        dmm 11y ago
        That makes sense, thanks
      - fredkbloggs 11y ago
        TLDR: This issue is way more complicated than you think, and the linked article is not very helpful in understanding it. It's fair to say that ZFS is as good as or better than anything else out there for any given storage topology and error case, but using ZFS in a redundant pool configuration is still necessary for universally good results. As usual with a technical subject, one has to be a lot more specific to get to the truth, both in describing the problem and identifying the exact behavior of software in response. "Disk failure" covers a lot of ground, and the article you linked doesn't narrow it down all that much. User data is stored on disk (or other media) along with a large number of ECC bits; the vendors know that bits rot on disk or in flash, and this allows transparent correction of most errors. So bits rot all the time and you normally never notice it; the rot is corrected by the disk firmware before any data is ever returned. Of greater interest are sectors that are repeatedly read in a way that they have so many errors as to be uncorrectable, which is probably what the author of that article intended to address. A disk that reliably fails a request for such a sector produces an error visible to the OS (or off-CPU RAID controller if in use), so that it can be tried against a mirror or computed from parity as appropriate. This case produces much the same result no matter what RAID or filesystem technology you use: a very slow read that eventually returns correct data. Other things may happen, too, like incrementing fault counters, sparing out the disk, rewriting the bad block (onto the same or a different location on the same or another disk), etc. These are dependent on the disk firmware and RAID or volume management implementation, and it's worth noting them because they greatly affect what happens the next time you try to read the same block and ultimately how likely you are to lose data to these defects. However, the more interesting cases (which the ZFS authors understand very well but were not specifically described in this article because its author does not appear to understand how disks work) are those in which the wrong data is returned. Contrary to the author's idea, this is usually not caused by media errors as those are almost always detected by ECC firmware. Instead, they are caused by bugs in the disk firmware, or even more commonly the OS, such as enabling a disk's write cache and then not flushing it to close the filesystem transaction. As another example, some software RAID implementations (and/or filesystems) by design require a nonvolatile cache for correctness, especially if parity RAID is used, but many machines do not have one installed or configured for use by software. Such problems cannot be corrected at read time; it's too late. There are of course many other causes of such "phantom writes" and all manner of other firmware bugs, and it's important to characterize them more fully. For example, sometimes a re-read of the same block will return correct data, assuming it was written in the first place, and sometimes it won't (even better, sometimes whether it does or not depends on transient and unobservable state!). Regardless of the underlying cause, ZFS can protect the user application from being given incorrect data, assuming the corruption is not happening within the computer itself, such as may be caused by an OS bug, non-ECC DRAM, or a bad CPU. If you use RAID (mirrors or parity) with ZFS, it can often return the correct data instead, and fix up the problem in the background as well as keeping track of these problems so that persistently or severely faulty disks can be condemned. Even if you do not use RAID, ZFS will (possibly after various retries) turn this into something that looks like an unrecoverable media error; the application gets EIO. Of course, in that case your data is still lost (unless you elected to enable ditto blocks for user data...). There's plenty more complexity here if you delve down into all the specific failure modes. Disks have an awful lot of them, and doing the right thing, or even agreeing on what the right thing is, for every one is difficult. Do you fail and spare out the disk? Do you count the errors and fail it after N of them? Do you rewrite the block, and if so at what address? Do you retry? How many times, and for how long? What if the disk returns correct data but only after a long time (usually caused by internal firmware retries)? And so on. "Bit rot", i.e., silent bit flips within underlying media, may be real, but it's not directly observable by software and it's only the least interesting tip of the disk failure iceberg.
cpach 11y ago
FreeNAS looks interesting, but I found the hardware requirements a bit prohibitive for home use. Renting storage in the ”cloud” feels more cost-effective.
- gh02t 11y ago
  The hardware requirements are sometimes exaggerated. 8GB (non-ecc) memory on a cheap processor is fine for a home user with FreeNAS. Whether or not it's cheaper depends on your use. Comparing to something like Glacier, FreeNAS is definitely cheaper if you need to store more than a couple hundred GB and/or access it frequently.
  - Laforet 11y ago
    You are quite right that 8GB memory and a cheap processor is good enough for home use. However I thought the consensus is that one cannot skip ECC with ZFS because a silent bitflip can corrupt an entire zpool. In any case the extra cost is minimal so why not.
    - gh02t 11y ago
      Running without ECC is no worse than doing so with any other filesystem [as far as I understand]. The thing about corrupting the entire zpool looks to be a bit of an urban legend (http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-yo...). Flipping bits can of course cause problems, but doing a scrub after a bit is flipped in memory doesn't hose your whole pool. There's been endless discussion and flame wars on the topic, but I've never once heard someone who actually had it happen (because it can't really happen). You can lose some data due to bit flips, you won't however lose everything in the whole volume like some people claim (unless you have something like an entire chip on the memory fail, then you're definitely screwed but neither ECC nor a different FS will help you there). ECC is a good idea and it's not hugely expensive, but there is one very big difference. That is, most people who might want to build a NAS have a decent bit of spare hardware lying around that they might be able to repurpose. That hardware is probably an old desktop machine and doesn't have ECC, so if you insist on using ECC then you can't use that old machine and have to buy a new mobo+RAM at least. I've been running ZFS for years without ECC doing weekly scrubs and never once destroyed anything. Edit: the counter-argument/source of the theory that you absolutely need ECC is primarily this post https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-ram-and-zfs.15449/ https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-... if you want to the other side for yourself.

I've been using it at home via FreeNAS after losing a bunch of old files to bitrot.

by zf00002 11y ago

I've been using it at home via FreeNAS after losing a bunch of old files to bitrot.

StavrosK 11y ago

How would ZFS help there? If you don't scrub, you lose files whatever fs you use.

abrookewood 11y ago
He never said he didn't scrub ... assuming he did (and he held 2+ copies), then the files are restored. Even if he only had a single copy, ZFS would at least tell you what files are corrupt, allowing you to restore them from backup.
- StavrosK 11y ago
  I guess that's true, I was thinking about the case of the redundant array (otherwise the files wouldn't be "lost").
  - dmm 11y ago
    Bitrot still affects traditional RAID. > As much as it affects ZFS, that's my point. That's not true. RAID protects you against disk failure but bitrot is caused by unrecoverable read failures. Check out this ars article: http://arstechnica.com/information-technology/2014/01/bitrot-and-atomic-cows-inside-next-gen-filesystems/ http://arstechnica.com/information-technology/2014/01/bitrot...
    - StavrosK 11y ago
      As much as it affects ZFS, that's my point.
    - ryao 11y ago
      RAID can handle unrecoverable read failures. What it cannot handle is silent corruption where bad data is returned as if it were the correct data.
      - dmm 11y ago
        That makes sense, thanks
    - fredkbloggs 11y ago
      TLDR: This issue is way more complicated than you think, and the linked article is not very helpful in understanding it. It's fair to say that ZFS is as good as or better than anything else out there for any given storage topology and error case, but using ZFS in a redundant pool configuration is still necessary for universally good results. As usual with a technical subject, one has to be a lot more specific to get to the truth, both in describing the problem and identifying the exact behavior of software in response. "Disk failure" covers a lot of ground, and the article you linked doesn't narrow it down all that much. User data is stored on disk (or other media) along with a large number of ECC bits; the vendors know that bits rot on disk or in flash, and this allows transparent correction of most errors. So bits rot all the time and you normally never notice it; the rot is corrected by the disk firmware before any data is ever returned. Of greater interest are sectors that are repeatedly read in a way that they have so many errors as to be uncorrectable, which is probably what the author of that article intended to address. A disk that reliably fails a request for such a sector produces an error visible to the OS (or off-CPU RAID controller if in use), so that it can be tried against a mirror or computed from parity as appropriate. This case produces much the same result no matter what RAID or filesystem technology you use: a very slow read that eventually returns correct data. Other things may happen, too, like incrementing fault counters, sparing out the disk, rewriting the bad block (onto the same or a different location on the same or another disk), etc. These are dependent on the disk firmware and RAID or volume management implementation, and it's worth noting them because they greatly affect what happens the next time you try to read the same block and ultimately how likely you are to lose data to these defects. However, the more interesting cases (which the ZFS authors understand very well but were not specifically described in this article because its author does not appear to understand how disks work) are those in which the wrong data is returned. Contrary to the author's idea, this is usually not caused by media errors as those are almost always detected by ECC firmware. Instead, they are caused by bugs in the disk firmware, or even more commonly the OS, such as enabling a disk's write cache and then not flushing it to close the filesystem transaction. As another example, some software RAID implementations (and/or filesystems) by design require a nonvolatile cache for correctness, especially if parity RAID is used, but many machines do not have one installed or configured for use by software. Such problems cannot be corrected at read time; it's too late. There are of course many other causes of such "phantom writes" and all manner of other firmware bugs, and it's important to characterize them more fully. For example, sometimes a re-read of the same block will return correct data, assuming it was written in the first place, and sometimes it won't (even better, sometimes whether it does or not depends on transient and unobservable state!). Regardless of the underlying cause, ZFS can protect the user application from being given incorrect data, assuming the corruption is not happening within the computer itself, such as may be caused by an OS bug, non-ECC DRAM, or a bad CPU. If you use RAID (mirrors or parity) with ZFS, it can often return the correct data instead, and fix up the problem in the background as well as keeping track of these problems so that persistently or severely faulty disks can be condemned. Even if you do not use RAID, ZFS will (possibly after various retries) turn this into something that looks like an unrecoverable media error; the application gets EIO. Of course, in that case your data is still lost (unless you elected to enable ditto blocks for user data...). There's plenty more complexity here if you delve down into all the specific failure modes. Disks have an awful lot of them, and doing the right thing, or even agreeing on what the right thing is, for every one is difficult. Do you fail and spare out the disk? Do you count the errors and fail it after N of them? Do you rewrite the block, and if so at what address? Do you retry? How many times, and for how long? What if the disk returns correct data but only after a long time (usually caused by internal firmware retries)? And so on. "Bit rot", i.e., silent bit flips within underlying media, may be real, but it's not directly observable by software and it's only the least interesting tip of the disk failure iceberg.

cpach 11y ago

FreeNAS looks interesting, but I found the hardware requirements a bit prohibitive for home use. Renting storage in the ”cloud” feels more cost-effective.

gh02t 11y ago
The hardware requirements are sometimes exaggerated. 8GB (non-ecc) memory on a cheap processor is fine for a home user with FreeNAS. Whether or not it's cheaper depends on your use. Comparing to something like Glacier, FreeNAS is definitely cheaper if you need to store more than a couple hundred GB and/or access it frequently.
- Laforet 11y ago
  You are quite right that 8GB memory and a cheap processor is good enough for home use. However I thought the consensus is that one cannot skip ECC with ZFS because a silent bitflip can corrupt an entire zpool. In any case the extra cost is minimal so why not.
  - gh02t 11y ago
    Running without ECC is no worse than doing so with any other filesystem [as far as I understand]. The thing about corrupting the entire zpool looks to be a bit of an urban legend (http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-yo...). Flipping bits can of course cause problems, but doing a scrub after a bit is flipped in memory doesn't hose your whole pool. There's been endless discussion and flame wars on the topic, but I've never once heard someone who actually had it happen (because it can't really happen). You can lose some data due to bit flips, you won't however lose everything in the whole volume like some people claim (unless you have something like an entire chip on the memory fail, then you're definitely screwed but neither ECC nor a different FS will help you there). ECC is a good idea and it's not hugely expensive, but there is one very big difference. That is, most people who might want to build a NAS have a decent bit of spare hardware lying around that they might be able to repurpose. That hardware is probably an old desktop machine and doesn't have ECC, so if you insist on using ECC then you can't use that old machine and have to buy a new mobo+RAM at least. I've been running ZFS for years without ECC doing weekly scrubs and never once destroyed anything. Edit: the counter-argument/source of the theory that you absolutely need ECC is primarily this post https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-ram-and-zfs.15449/ https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-... if you want to the other side for yourself.