6 ms·

I hate this headline and wince every time I see it, even the article quotes > There's nothing special about ZFS that requires/encourages the use of ECC RAM mor

I hate this headline and wince every time I see it, even the article quotes > There's nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem. If you use UFS, EXT, NTFS, btrfs, etc without ECC RAM, you are just as much at risk as if you used ZFS without ECC RAM. I would simply say: if you love your data, use ECC RAM. Additionally, use a filesystem that checksums your data, such as ZFS. And goes on to say > I have nothing to substantiate this, but my thinking is that since ZFS is a way more advanced and complex file system, it may be more susceptible to the adverse effects of bad memory, compared to legacy file systems.

PragmaticPulp 4y ago
I’m actually fascinated by how much this post became accepted canon over the years. The truth is that everyone should want to use ECC wherever data integrity is a priority, regardless of whether or not they’re using ZFS. The conjecture about ZFS being uniquely vulnerable has been debunked over and over again. Maybe there was something different back in 2014 when this was written, but even if that was the case it’s not a good idea to use such old blog posts full of admitted conjecture to anchor your technology understandings.
- wmf 4y ago
  People love clever contrarian takes. A thing that sounds good is actually bad!
- colechristensen 4y ago
  Eh, the chance that data I care about will get corrupted in a way that matters is very small. ECC also does not eliminate errors, just reduces them by some constant proportion. Something quite rare becoming say, 10 times less likely doesn't really effect my decisions much. If you're using ECC you can make the exact same argument that you need another bit of parity with the same justifications of making something rare, rarer. Of the things than can go wrong with my data, small corruptions from memory bit flips are far down on the risks to worry about. If you're Google (you're not Google) then the scale of errors becomes a choice of optimization.
  - RL_Quine 4y ago
    I've seen many ECC log lines on machines before, my wifi routers have ECC memory and they have seen at least one fault each. It isn't a one in a million sort of risk, it's just that most systems don't run with ECC, and maybe that bit flip just causes a crash or instability that's chalked up to software bugs. Especially as memory densities continue to skyrocket I would expect to see errors becoming extremely common place. It's some sort of variation of the Toupee Fallacy, you aren't aware of the frequency of bit errors because you have absolutely no way of knowing when they happen, and what "random" errors were caused by them.
    - colechristensen 4y ago
      It’s not that errors don’t happen it’s that they are rare and the surface area of them affecting something important in an important way are incredibly rare. I don’t care about triggering an occasional bug or a bit flip in a video stream, these things are inconsequential. If a bit gets flipped in a photo it’ll be a very slightly different photo. A piece of text will be slightly different… if either had the luck to be in memory ready to write when hit, in dozens of gigabytes of memory. The most likely consequence of a bit flip is nothing at all happening because it hit unallocated memory. The next most likely consequence is nothing as it hit a piece of program memory that has no effect on operation and gets overwritten in moments. ECC zealots are really excited about preventing something that just doesn’t happen all that often: actual data corruption with permanent significant effects.
      - RL_Quine 4y ago
        My point was that you don't know when corruption has had an effect, because you're eschewing a method of detecting it.
    - matheusmoreira 4y ago
      > my wifi routers have ECC memory This is really weird. How come even routers have ECC memory while our actual computers don't? Why can't all memory just be ECC memory? I don't even see the point of non-ECC memory. Everything has error correction built-in: hard drives, SSDs, even optical media. Yet RAM doesn't?
- Wowfunhappy 4y ago
  > I’m actually fascinated by how much this post became accepted canon over the years. I think the misunderstanding goes something like this. I'm pretty sure all three of the below statements are true: • People who prioritize data integrity are more likely to be using ZFS. • People who prioritize data integrity should use ECC memory. • Ergo, if you are using ZFS, you should probably be using ECC memory. However, it doesn't naturally follow that because you are using ZFS, you should be using ECC memory, nor that people without ECC memory should not use ZFS. This is unintuitive.
  - chasil 4y ago
    I have an Oracle Linux PC, running everything in btrfs. I use a usb-attached zfs raid-1 for my important storage. I have to run "zfs-fuse" before I mount it. What I know about ecc comes from this: https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=26303271#p26303271 https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=... I have never scrubbed my zfs vault, but I have scrubbed my btrfs drives. I should get something with ecc. Wow, do I have a lot of stuff.
    - Wowfunhappy 4y ago
      Actually, while we're here, I would love to know more about the ZFS_DEBUG_MODIFY flag at that link. I've come across that post before and have always been wondering if I should be using the flag, since I can't practically use ECC. But the flag has debug in the name and I've never been able to find more information about it.
    - michaelmrose 4y ago
      Why on earth are you running oracle linux or zfs-fuse? I have no idea why one would use a 2006 Google summer of code project instead of the official openzfs project which provides an actual kernel module that would be several times faster and likely safer.
    - michaelmrose 4y ago
      I would like to add that running a zfs raid array without ever scrubbing is asking for data loss ask Linus from Linus Tech Tips how he lost data repeatedly.
- bsder 4y ago
  > Maybe there was something different back in 2014 when this was written The problem was the same even in 2014: ZFS lets you know how shitty your hardware is. And, just like 2014, people would rather shoot the messenger because they don't have any power to force companies to make better hardware.
- p_l 4y ago
  There was a concerted effort to push the claim everywhere close to FreeNAS, to the point that another "community canon" was that FreeNAS is behind it, as they got money from selling you "certified NAS boxes with ECC memory". The only things that really happened was that a) ZFS exposes how failure prone hardware is due to pervasive checksumming b) ZFS came from culture (and project goals) that prioritized data safety - as such, it was possibly first contact for many with strong recommendations about ECC memory
  - kennend3 4y ago
    I remember using FreeNAS many years ago and would regularity check out the forums. The amount of people rehashing this "you must use ECC with ZFS" was insane. One guy seemed to have made it his life's mission to publish this same story over and over again.
- dizhn 4y ago
  Another myth that still lives on is that you need 1gb of ram for every 1tb of disk on zfs.
  - GekkePrutser 4y ago
    ZFS dedup made that a thing but it's almost never used by consumers.
kurlberg 4y ago
This has been discussed on HN some times before. User xornot looked at the zfs source code and debunked it, for more details see https://news.ycombinator.com/item?id=14207520 https://news.ycombinator.com/item?id=14207520
- deleted 4y ago
  [deleted]
yjftsjthsd-h 4y ago
Actually, I would have expected that it was more important to use ECC with a filesystem that lacks data checksums, since those should at least catch corruption.
- dathinab 4y ago
  ECC is not about disk corruption, while the checksums are about disk corruption ECC is corruption of the parts in memory, like caches of data, data when it's in the process of being compressed or encrypted, some management data always cached in memory etc. In the end not having ECC can affect your system in such abstruse ways that IMHO it probably doesn't make any relevant difference if you use zfs or e.g. ext4. Tbh it's kind absurd that ECC is not the standard for any non-"cheap" PC/Laptop. EDIT: Checksums can still potentially help in the unlikely case that the RAM error is a bitflip in some loaded offset/ptr referring "into" the HDD/SSD storage (e.g. position of some file on disk).
- Xamayon 4y ago
  The good thing about filesystems which have no error checking is that they won't generally go back and corrupt the old, idle, already written data on disk when they have a memory issue. If you run a scrub/whatever to check validity of old data and your system has a new memory issue, it could destroy everything very quickly even without new file writes. With a less feature full filesystem your data corruption could still be quite bad, but it would have a better chance for recovery as even if the filesystem is severely damaged, the file data would be (mostly) untouched. ECC helps prevent both of these cases, it's really too bad it's so hard/expensive to use outside of server hardware.
  - Wowfunhappy 4y ago
    > If you run a scrub/whatever to check validity of old data and your system has a new memory issue, it could destroy everything very quickly even without new file writes. No, it won't. https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-y... > Let’s assume that we have RAM that not only isn’t working 100% properly, but is actively goddamn evil and trying its naive but enthusiastic best to specifically kill your data during a scrub. First, you read a block. This block is good. It is perfectly good data written to a perfectly good disk with a perfectly matching checksum. But that block is read into evil RAM, and the evil RAM flips some bits. Perhaps those bits are in the data itself, or perhaps those bits are in the checksum. Either way, your perfectly good block now does not appear to match its checksum, and since we’re scrubbing, ZFS will attempt to actually repair the “bad” block on disk. Uh-oh! What now? > Next, you read [a copy of the same block from another disk]. Now, if your evil RAM leaves this block alone, ZFS will see that the second copy matches its checksum, and so it will overwrite the first block with the same data it had originally – no data was lost here, just a few wasted disk cycles. OK. But what if your evil RAM flips a bit in the second copy? Since it doesn’t match the checksum either, ZFS doesn’t overwrite anything. It logs an unrecoverable data error for that block, and leaves both copies untouched on disk.
michaelmrose 4y ago
I have nothing to substantiate this is another way of saying "I'm making this up" but somehow think my intuition is worth more than the actual knowledge of the experts who actually did the work. It's as suspect as I have nothing to substantiate it but my kid caught autism after that shot. It's so badly reasoned it calls into question the authors judgment on any technical topic because it says his basic tools for reasoning are lacking. 99.9% of data loss occurs because of well understood but difficult to fully mitigate problems. There is no reason to believe zfs is more susceptible to bad memory and it's not exactly new having begun development 21 years ago.
Brian_K_White 4y ago
It's not up for debate or interpretation that zfs relies on ram more than most other filesystems. They all use ram, because everything uses ram, and so that much is a baseline that cancels out between anything and anything else. And zfs uses more and relies on it more than others, above that baseline.
- p_l 4y ago
  Everything that uses linux pagecache as interface (i.e. everything not ZFS on linux) is way, way more interleaved with memory usage than ZFS, with some extra gotchas due to how pagecache works by mapping disk and memory 1:1 (great simplification, but unless you go out of your way the FS implementation of write()/read() is never used - instead the the same operations as with mmap() are called and write/read calls copy from memory.)

PragmaticPulp 4y ago

I’m actually fascinated by how much this post became accepted canon over the years. The truth is that everyone should want to use ECC wherever data integrity is a priority, regardless of whether or not they’re using ZFS. The conjecture about ZFS being uniquely vulnerable has been debunked over and over again. Maybe there was something different back in 2014 when this was written, but even if that was the case it’s not a good idea to use such old blog posts full of admitted conjecture to anchor your technology understandings.

wmf 4y ago
People love clever contrarian takes. A thing that sounds good is actually bad!
colechristensen 4y ago
Eh, the chance that data I care about will get corrupted in a way that matters is very small. ECC also does not eliminate errors, just reduces them by some constant proportion. Something quite rare becoming say, 10 times less likely doesn't really effect my decisions much. If you're using ECC you can make the exact same argument that you need another bit of parity with the same justifications of making something rare, rarer. Of the things than can go wrong with my data, small corruptions from memory bit flips are far down on the risks to worry about. If you're Google (you're not Google) then the scale of errors becomes a choice of optimization.
- RL_Quine 4y ago
  I've seen many ECC log lines on machines before, my wifi routers have ECC memory and they have seen at least one fault each. It isn't a one in a million sort of risk, it's just that most systems don't run with ECC, and maybe that bit flip just causes a crash or instability that's chalked up to software bugs. Especially as memory densities continue to skyrocket I would expect to see errors becoming extremely common place. It's some sort of variation of the Toupee Fallacy, you aren't aware of the frequency of bit errors because you have absolutely no way of knowing when they happen, and what "random" errors were caused by them.
  - colechristensen 4y ago
    It’s not that errors don’t happen it’s that they are rare and the surface area of them affecting something important in an important way are incredibly rare. I don’t care about triggering an occasional bug or a bit flip in a video stream, these things are inconsequential. If a bit gets flipped in a photo it’ll be a very slightly different photo. A piece of text will be slightly different… if either had the luck to be in memory ready to write when hit, in dozens of gigabytes of memory. The most likely consequence of a bit flip is nothing at all happening because it hit unallocated memory. The next most likely consequence is nothing as it hit a piece of program memory that has no effect on operation and gets overwritten in moments. ECC zealots are really excited about preventing something that just doesn’t happen all that often: actual data corruption with permanent significant effects.
    - RL_Quine 4y ago
      My point was that you don't know when corruption has had an effect, because you're eschewing a method of detecting it.
  - matheusmoreira 4y ago
    > my wifi routers have ECC memory This is really weird. How come even routers have ECC memory while our actual computers don't? Why can't all memory just be ECC memory? I don't even see the point of non-ECC memory. Everything has error correction built-in: hard drives, SSDs, even optical media. Yet RAM doesn't?
Wowfunhappy 4y ago
> I’m actually fascinated by how much this post became accepted canon over the years. I think the misunderstanding goes something like this. I'm pretty sure all three of the below statements are true: • People who prioritize data integrity are more likely to be using ZFS. • People who prioritize data integrity should use ECC memory. • Ergo, if you are using ZFS, you should probably be using ECC memory. However, it doesn't naturally follow that because you are using ZFS, you should be using ECC memory, nor that people without ECC memory should not use ZFS. This is unintuitive.
- chasil 4y ago
  I have an Oracle Linux PC, running everything in btrfs. I use a usb-attached zfs raid-1 for my important storage. I have to run "zfs-fuse" before I mount it. What I know about ecc comes from this: https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=26303271#p26303271 https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=... I have never scrubbed my zfs vault, but I have scrubbed my btrfs drives. I should get something with ecc. Wow, do I have a lot of stuff.
  - Wowfunhappy 4y ago
    Actually, while we're here, I would love to know more about the ZFS_DEBUG_MODIFY flag at that link. I've come across that post before and have always been wondering if I should be using the flag, since I can't practically use ECC. But the flag has debug in the name and I've never been able to find more information about it.
  - michaelmrose 4y ago
    Why on earth are you running oracle linux or zfs-fuse? I have no idea why one would use a 2006 Google summer of code project instead of the official openzfs project which provides an actual kernel module that would be several times faster and likely safer.
  - michaelmrose 4y ago
    I would like to add that running a zfs raid array without ever scrubbing is asking for data loss ask Linus from Linus Tech Tips how he lost data repeatedly.
bsder 4y ago
> Maybe there was something different back in 2014 when this was written The problem was the same even in 2014: ZFS lets you know how shitty your hardware is. And, just like 2014, people would rather shoot the messenger because they don't have any power to force companies to make better hardware.
p_l 4y ago
There was a concerted effort to push the claim everywhere close to FreeNAS, to the point that another "community canon" was that FreeNAS is behind it, as they got money from selling you "certified NAS boxes with ECC memory". The only things that really happened was that a) ZFS exposes how failure prone hardware is due to pervasive checksumming b) ZFS came from culture (and project goals) that prioritized data safety - as such, it was possibly first contact for many with strong recommendations about ECC memory
- kennend3 4y ago
  I remember using FreeNAS many years ago and would regularity check out the forums. The amount of people rehashing this "you must use ECC with ZFS" was insane. One guy seemed to have made it his life's mission to publish this same story over and over again.
dizhn 4y ago
Another myth that still lives on is that you need 1gb of ram for every 1tb of disk on zfs.
- GekkePrutser 4y ago
  ZFS dedup made that a thing but it's almost never used by consumers.

kurlberg 4y ago

This has been discussed on HN some times before. User xornot looked at the zfs source code and debunked it, for more details see https://news.ycombinator.com/item?id=14207520 https://news.ycombinator.com/item?id=14207520

deleted 4y ago
[deleted]

yjftsjthsd-h 4y ago

Actually, I would have expected that it was more important to use ECC with a filesystem that lacks data checksums, since those should at least catch corruption.

dathinab 4y ago
ECC is not about disk corruption, while the checksums are about disk corruption ECC is corruption of the parts in memory, like caches of data, data when it's in the process of being compressed or encrypted, some management data always cached in memory etc. In the end not having ECC can affect your system in such abstruse ways that IMHO it probably doesn't make any relevant difference if you use zfs or e.g. ext4. Tbh it's kind absurd that ECC is not the standard for any non-"cheap" PC/Laptop. EDIT: Checksums can still potentially help in the unlikely case that the RAM error is a bitflip in some loaded offset/ptr referring "into" the HDD/SSD storage (e.g. position of some file on disk).
Xamayon 4y ago
The good thing about filesystems which have no error checking is that they won't generally go back and corrupt the old, idle, already written data on disk when they have a memory issue. If you run a scrub/whatever to check validity of old data and your system has a new memory issue, it could destroy everything very quickly even without new file writes. With a less feature full filesystem your data corruption could still be quite bad, but it would have a better chance for recovery as even if the filesystem is severely damaged, the file data would be (mostly) untouched. ECC helps prevent both of these cases, it's really too bad it's so hard/expensive to use outside of server hardware.
- Wowfunhappy 4y ago
  > If you run a scrub/whatever to check validity of old data and your system has a new memory issue, it could destroy everything very quickly even without new file writes. No, it won't. https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-y... > Let’s assume that we have RAM that not only isn’t working 100% properly, but is actively goddamn evil and trying its naive but enthusiastic best to specifically kill your data during a scrub. First, you read a block. This block is good. It is perfectly good data written to a perfectly good disk with a perfectly matching checksum. But that block is read into evil RAM, and the evil RAM flips some bits. Perhaps those bits are in the data itself, or perhaps those bits are in the checksum. Either way, your perfectly good block now does not appear to match its checksum, and since we’re scrubbing, ZFS will attempt to actually repair the “bad” block on disk. Uh-oh! What now? > Next, you read [a copy of the same block from another disk]. Now, if your evil RAM leaves this block alone, ZFS will see that the second copy matches its checksum, and so it will overwrite the first block with the same data it had originally – no data was lost here, just a few wasted disk cycles. OK. But what if your evil RAM flips a bit in the second copy? Since it doesn’t match the checksum either, ZFS doesn’t overwrite anything. It logs an unrecoverable data error for that block, and leaves both copies untouched on disk.

michaelmrose 4y ago

I have nothing to substantiate this is another way of saying "I'm making this up" but somehow think my intuition is worth more than the actual knowledge of the experts who actually did the work. It's as suspect as I have nothing to substantiate it but my kid caught autism after that shot. It's so badly reasoned it calls into question the authors judgment on any technical topic because it says his basic tools for reasoning are lacking. 99.9% of data loss occurs because of well understood but difficult to fully mitigate problems. There is no reason to believe zfs is more susceptible to bad memory and it's not exactly new having begun development 21 years ago.

Brian_K_White 4y ago

It's not up for debate or interpretation that zfs relies on ram more than most other filesystems. They all use ram, because everything uses ram, and so that much is a baseline that cancels out between anything and anything else. And zfs uses more and relies on it more than others, above that baseline.

p_l 4y ago
Everything that uses linux pagecache as interface (i.e. everything not ZFS on linux) is way, way more interleaved with memory usage than ZFS, with some extra gotchas due to how pagecache works by mapping disk and memory 1:1 (great simplification, but unless you go out of your way the FS implementation of write()/read() is never used - instead the the same operations as with mmap() are called and write/read calls copy from memory.)