4 ms·

I’m actually fascinated by how much this post became accepted canon over the years. The truth is that everyone should want to use ECC wherever data integrity i

I’m actually fascinated by how much this post became accepted canon over the years. The truth is that everyone should want to use ECC wherever data integrity is a priority, regardless of whether or not they’re using ZFS. The conjecture about ZFS being uniquely vulnerable has been debunked over and over again. Maybe there was something different back in 2014 when this was written, but even if that was the case it’s not a good idea to use such old blog posts full of admitted conjecture to anchor your technology understandings.

wmf 4y ago
People love clever contrarian takes. A thing that sounds good is actually bad!
colechristensen 4y ago
Eh, the chance that data I care about will get corrupted in a way that matters is very small. ECC also does not eliminate errors, just reduces them by some constant proportion. Something quite rare becoming say, 10 times less likely doesn't really effect my decisions much. If you're using ECC you can make the exact same argument that you need another bit of parity with the same justifications of making something rare, rarer. Of the things than can go wrong with my data, small corruptions from memory bit flips are far down on the risks to worry about. If you're Google (you're not Google) then the scale of errors becomes a choice of optimization.
- RL_Quine 4y ago
  I've seen many ECC log lines on machines before, my wifi routers have ECC memory and they have seen at least one fault each. It isn't a one in a million sort of risk, it's just that most systems don't run with ECC, and maybe that bit flip just causes a crash or instability that's chalked up to software bugs. Especially as memory densities continue to skyrocket I would expect to see errors becoming extremely common place. It's some sort of variation of the Toupee Fallacy, you aren't aware of the frequency of bit errors because you have absolutely no way of knowing when they happen, and what "random" errors were caused by them.
  - colechristensen 4y ago
    It’s not that errors don’t happen it’s that they are rare and the surface area of them affecting something important in an important way are incredibly rare. I don’t care about triggering an occasional bug or a bit flip in a video stream, these things are inconsequential. If a bit gets flipped in a photo it’ll be a very slightly different photo. A piece of text will be slightly different… if either had the luck to be in memory ready to write when hit, in dozens of gigabytes of memory. The most likely consequence of a bit flip is nothing at all happening because it hit unallocated memory. The next most likely consequence is nothing as it hit a piece of program memory that has no effect on operation and gets overwritten in moments. ECC zealots are really excited about preventing something that just doesn’t happen all that often: actual data corruption with permanent significant effects.
    - RL_Quine 4y ago
      My point was that you don't know when corruption has had an effect, because you're eschewing a method of detecting it.
  - matheusmoreira 4y ago
    > my wifi routers have ECC memory This is really weird. How come even routers have ECC memory while our actual computers don't? Why can't all memory just be ECC memory? I don't even see the point of non-ECC memory. Everything has error correction built-in: hard drives, SSDs, even optical media. Yet RAM doesn't?
    - adrian_b 4y ago
      Using memory without ECC on the so-called "consumer" devices, in opposition with the "professional" devices, a.k.a. servers and workstations, has been caused by Intel, from 1994, when they have introduced the second generation of Pentium CPUs, to 1995, when they have introduced Pentium Pro. (The 1st generation of Pentium, the 60/66 MHz models, had been introduced in 1993, while the 2nd generation, the 90/100 MHz models, introduced in 1994, used a different socket and different motherboards.) Unlike with the previous Intel CPUs, Intel had the idea to introduce a market segmentation between Pentium and Pentium Pro (for the successors of Pentium Pro, Intel created the Xeon brand, a couple of years later). Therefore, Intel has retained the ability to detect and or correct memory errors only in the Pentium Pro chipsets (at that time the memory controllers were in the so called northbridge, a part of the chipset, and not in the CPU), while removing this feature from the Triton chipset intended for Pentium. While Intel has initiated this market segmentation policy, between "amateurs" and "professionals" (unlike IBM, which since the first IBM PC had always included memory error detection via parity), both the memory module vendors and the makers of other CPUs have been also happy to follow this policy, because it allows them to extract much more money from the knowledgeable users than the cost of adding ECC, while also having larger profit margins when selling to naive users, because the elimination of ECC/parity has never caused any price reduction. When this feature was removed, all the new motherboards and modules without parity/ECC had the same prices as the previous models with error detection (for many years, the memory errors were detected via parity, but then the memory controllers were improved, so that when using the same number of extra bits, i.e. the same memory modules and motherboards as for parity, single errors could be corrected, not only detected).
- 10987654321 4y ago
  As someone who doesn't use ECC memory for my NAS (I'm not google) I have experienced many bitflips / corruptions for some sets of photos, especially when using a NAS with what turned out to be bad ram sticks (which was recognized after switching to using ZFS and raid 10, I don't know if this saved any of my data but the setup at least made me recognize that I had a problem).
- adrian_b 4y ago
  ECC does not reduce errors only 10 times, but by several orders of magnitude. The frequency of DRAM errors is proportional with the amount of memory installed in a computer. For smaller quantities of memory, e.g. 16 GB or 32 GB, the frequency might be of one error every few months. With ECC, the mean time between errors is likely to become longer than the lifetime of the computer. Intel, who has initiated this absurdity of convincing the naive customers that it is OK to buy computers that may make mistakes from time to time, has succeeded to get away with it only due to the huge amount of software bugs, which have made the computer users believe that whenever a computer crashes, or some corrupted data is discovered, it is more likely that the cause has been a software bug and not a hardware defect. The integrated circuits made with modern CMOS processes, with very small devices, have a non-negligible aging rate. Because of that, memory modules that have been used for many years start at some point to have much more frequent errors than when new. When you have ECC, such memory modules are immediately detected and they can be replaced before causing some irremediable data corruption. This helped me a lot a few times.
Wowfunhappy 4y ago
> I’m actually fascinated by how much this post became accepted canon over the years. I think the misunderstanding goes something like this. I'm pretty sure all three of the below statements are true: • People who prioritize data integrity are more likely to be using ZFS. • People who prioritize data integrity should use ECC memory. • Ergo, if you are using ZFS, you should probably be using ECC memory. However, it doesn't naturally follow that because you are using ZFS, you should be using ECC memory, nor that people without ECC memory should not use ZFS. This is unintuitive.
- chasil 4y ago
  I have an Oracle Linux PC, running everything in btrfs. I use a usb-attached zfs raid-1 for my important storage. I have to run "zfs-fuse" before I mount it. What I know about ecc comes from this: https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=26303271#p26303271 https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=... I have never scrubbed my zfs vault, but I have scrubbed my btrfs drives. I should get something with ecc. Wow, do I have a lot of stuff.
  - Wowfunhappy 4y ago
    Actually, while we're here, I would love to know more about the ZFS_DEBUG_MODIFY flag at that link. I've come across that post before and have always been wondering if I should be using the flag, since I can't practically use ECC. But the flag has debug in the name and I've never been able to find more information about it.
  - michaelmrose 4y ago
    Why on earth are you running oracle linux or zfs-fuse? I have no idea why one would use a 2006 Google summer of code project instead of the official openzfs project which provides an actual kernel module that would be several times faster and likely safer.
  - michaelmrose 4y ago
    I would like to add that running a zfs raid array without ever scrubbing is asking for data loss ask Linus from Linus Tech Tips how he lost data repeatedly.
bsder 4y ago
> Maybe there was something different back in 2014 when this was written The problem was the same even in 2014: ZFS lets you know how shitty your hardware is. And, just like 2014, people would rather shoot the messenger because they don't have any power to force companies to make better hardware.
p_l 4y ago
There was a concerted effort to push the claim everywhere close to FreeNAS, to the point that another "community canon" was that FreeNAS is behind it, as they got money from selling you "certified NAS boxes with ECC memory". The only things that really happened was that a) ZFS exposes how failure prone hardware is due to pervasive checksumming b) ZFS came from culture (and project goals) that prioritized data safety - as such, it was possibly first contact for many with strong recommendations about ECC memory
- kennend3 4y ago
  I remember using FreeNAS many years ago and would regularity check out the forums. The amount of people rehashing this "you must use ECC with ZFS" was insane. One guy seemed to have made it his life's mission to publish this same story over and over again.
dizhn 4y ago
Another myth that still lives on is that you need 1gb of ram for every 1tb of disk on zfs.
- GekkePrutser 4y ago
  ZFS dedup made that a thing but it's almost never used by consumers.

wmf 4y ago

People love clever contrarian takes. A thing that sounds good is actually bad!

colechristensen 4y ago

Eh, the chance that data I care about will get corrupted in a way that matters is very small. ECC also does not eliminate errors, just reduces them by some constant proportion. Something quite rare becoming say, 10 times less likely doesn't really effect my decisions much. If you're using ECC you can make the exact same argument that you need another bit of parity with the same justifications of making something rare, rarer. Of the things than can go wrong with my data, small corruptions from memory bit flips are far down on the risks to worry about. If you're Google (you're not Google) then the scale of errors becomes a choice of optimization.

RL_Quine 4y ago
I've seen many ECC log lines on machines before, my wifi routers have ECC memory and they have seen at least one fault each. It isn't a one in a million sort of risk, it's just that most systems don't run with ECC, and maybe that bit flip just causes a crash or instability that's chalked up to software bugs. Especially as memory densities continue to skyrocket I would expect to see errors becoming extremely common place. It's some sort of variation of the Toupee Fallacy, you aren't aware of the frequency of bit errors because you have absolutely no way of knowing when they happen, and what "random" errors were caused by them.
- colechristensen 4y ago
  It’s not that errors don’t happen it’s that they are rare and the surface area of them affecting something important in an important way are incredibly rare. I don’t care about triggering an occasional bug or a bit flip in a video stream, these things are inconsequential. If a bit gets flipped in a photo it’ll be a very slightly different photo. A piece of text will be slightly different… if either had the luck to be in memory ready to write when hit, in dozens of gigabytes of memory. The most likely consequence of a bit flip is nothing at all happening because it hit unallocated memory. The next most likely consequence is nothing as it hit a piece of program memory that has no effect on operation and gets overwritten in moments. ECC zealots are really excited about preventing something that just doesn’t happen all that often: actual data corruption with permanent significant effects.
  - RL_Quine 4y ago
    My point was that you don't know when corruption has had an effect, because you're eschewing a method of detecting it.
- matheusmoreira 4y ago
  > my wifi routers have ECC memory This is really weird. How come even routers have ECC memory while our actual computers don't? Why can't all memory just be ECC memory? I don't even see the point of non-ECC memory. Everything has error correction built-in: hard drives, SSDs, even optical media. Yet RAM doesn't?
  - adrian_b 4y ago
    Using memory without ECC on the so-called "consumer" devices, in opposition with the "professional" devices, a.k.a. servers and workstations, has been caused by Intel, from 1994, when they have introduced the second generation of Pentium CPUs, to 1995, when they have introduced Pentium Pro. (The 1st generation of Pentium, the 60/66 MHz models, had been introduced in 1993, while the 2nd generation, the 90/100 MHz models, introduced in 1994, used a different socket and different motherboards.) Unlike with the previous Intel CPUs, Intel had the idea to introduce a market segmentation between Pentium and Pentium Pro (for the successors of Pentium Pro, Intel created the Xeon brand, a couple of years later). Therefore, Intel has retained the ability to detect and or correct memory errors only in the Pentium Pro chipsets (at that time the memory controllers were in the so called northbridge, a part of the chipset, and not in the CPU), while removing this feature from the Triton chipset intended for Pentium. While Intel has initiated this market segmentation policy, between "amateurs" and "professionals" (unlike IBM, which since the first IBM PC had always included memory error detection via parity), both the memory module vendors and the makers of other CPUs have been also happy to follow this policy, because it allows them to extract much more money from the knowledgeable users than the cost of adding ECC, while also having larger profit margins when selling to naive users, because the elimination of ECC/parity has never caused any price reduction. When this feature was removed, all the new motherboards and modules without parity/ECC had the same prices as the previous models with error detection (for many years, the memory errors were detected via parity, but then the memory controllers were improved, so that when using the same number of extra bits, i.e. the same memory modules and motherboards as for parity, single errors could be corrected, not only detected).
10987654321 4y ago
As someone who doesn't use ECC memory for my NAS (I'm not google) I have experienced many bitflips / corruptions for some sets of photos, especially when using a NAS with what turned out to be bad ram sticks (which was recognized after switching to using ZFS and raid 10, I don't know if this saved any of my data but the setup at least made me recognize that I had a problem).
adrian_b 4y ago
ECC does not reduce errors only 10 times, but by several orders of magnitude. The frequency of DRAM errors is proportional with the amount of memory installed in a computer. For smaller quantities of memory, e.g. 16 GB or 32 GB, the frequency might be of one error every few months. With ECC, the mean time between errors is likely to become longer than the lifetime of the computer. Intel, who has initiated this absurdity of convincing the naive customers that it is OK to buy computers that may make mistakes from time to time, has succeeded to get away with it only due to the huge amount of software bugs, which have made the computer users believe that whenever a computer crashes, or some corrupted data is discovered, it is more likely that the cause has been a software bug and not a hardware defect. The integrated circuits made with modern CMOS processes, with very small devices, have a non-negligible aging rate. Because of that, memory modules that have been used for many years start at some point to have much more frequent errors than when new. When you have ECC, such memory modules are immediately detected and they can be replaced before causing some irremediable data corruption. This helped me a lot a few times.

Wowfunhappy 4y ago

> I’m actually fascinated by how much this post became accepted canon over the years. I think the misunderstanding goes something like this. I'm pretty sure all three of the below statements are true: • People who prioritize data integrity are more likely to be using ZFS. • People who prioritize data integrity should use ECC memory. • Ergo, if you are using ZFS, you should probably be using ECC memory. However, it doesn't naturally follow that because you are using ZFS, you should be using ECC memory, nor that people without ECC memory should not use ZFS. This is unintuitive.

chasil 4y ago
I have an Oracle Linux PC, running everything in btrfs. I use a usb-attached zfs raid-1 for my important storage. I have to run "zfs-fuse" before I mount it. What I know about ecc comes from this: https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=26303271#p26303271 https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=... I have never scrubbed my zfs vault, but I have scrubbed my btrfs drives. I should get something with ecc. Wow, do I have a lot of stuff.
- Wowfunhappy 4y ago
  Actually, while we're here, I would love to know more about the ZFS_DEBUG_MODIFY flag at that link. I've come across that post before and have always been wondering if I should be using the flag, since I can't practically use ECC. But the flag has debug in the name and I've never been able to find more information about it.
- michaelmrose 4y ago
  Why on earth are you running oracle linux or zfs-fuse? I have no idea why one would use a 2006 Google summer of code project instead of the official openzfs project which provides an actual kernel module that would be several times faster and likely safer.
- michaelmrose 4y ago
  I would like to add that running a zfs raid array without ever scrubbing is asking for data loss ask Linus from Linus Tech Tips how he lost data repeatedly.

bsder 4y ago

> Maybe there was something different back in 2014 when this was written The problem was the same even in 2014: ZFS lets you know how shitty your hardware is. And, just like 2014, people would rather shoot the messenger because they don't have any power to force companies to make better hardware.

p_l 4y ago

There was a concerted effort to push the claim everywhere close to FreeNAS, to the point that another "community canon" was that FreeNAS is behind it, as they got money from selling you "certified NAS boxes with ECC memory". The only things that really happened was that a) ZFS exposes how failure prone hardware is due to pervasive checksumming b) ZFS came from culture (and project goals) that prioritized data safety - as such, it was possibly first contact for many with strong recommendations about ECC memory

kennend3 4y ago
I remember using FreeNAS many years ago and would regularity check out the forums. The amount of people rehashing this "you must use ECC with ZFS" was insane. One guy seemed to have made it his life's mission to publish this same story over and over again.

dizhn 4y ago

Another myth that still lives on is that you need 1gb of ram for every 1tb of disk on zfs.

GekkePrutser 4y ago
ZFS dedup made that a thing but it's almost never used by consumers.