6 ms·

I use ZFS on Linux for my file server. One of the nicest things about it is that the caching algos actually work - they're resilient to scans, so I can, for in

I use ZFS on Linux for my file server. One of the nicest things about it is that the caching algos actually work - they're resilient to scans, so I can, for instance, have an intensively used database file that remains in RAM, even while & after doing a linear scan over a large number of files (eg during rsync). With the standard Linux page replacement algos, linear reads will flush the stuff you're actually using out of the page cache. The fact that the caching algos are so good at keeping things in memory is why everyone gets hung up on using ECC RAM with it.

mrb 12y ago
This is not the reason why people recommend ECC. ZFS is so good at detecting corruption (CKSUM errors) that when it happens it is hard to tell what caused it: faulty RAM corrupting data before being written to disk or after being read from disk, or data corrupted on disk? ECC simply helps reduce a good chunk of corruption errors caused by faulty RAM. That said, in my experience there are a few tell-tale signs that RAM is the issue: when you see CKSUM errors poping up infrequently, and not being attributed to consistently the same drive(s).
- byuu 12y ago
  I haven't had any issues so far using four drives mirrored on ZFS, but the ECC issue certainly worries me. I'd love to run ECC RAM, but I'd have to buy a much more expensive processor, a much more expensive mainboard, and all of my RAM would have to be swapped for moderately more expensive replacements. And DDR3+ is not cheap. I'm curious though, if ECC consists of a ninth parity bit, is there any reason why a memory controller can't be designed that would, worst case, use every other (identical) stick just to get parity bit(s), as a BIOS configurable option? Sure it'd halve your RAM, or you'd pay a lot more in RAM costs, but not having to buy Xeon processors and mainboards, and getting to reuse your existing RAM, would be worth buying an extra stick of RAM in my opinion. It seems as processes keep getting smaller, and RAM sizes keep getting larger, that the effects of cosmic radiation are just going to keep getting worse. If we can't get desktop CPUs and mainboards to just switch to ECC, surely this would at least be a better-than-nothing option.
  - _ihaque 12y ago
    Most modern CPUs have the memory controller integrated on-die, so you'd need a different class of CPU -- and then you're right back to buying a Xeon.
    - rodgerd 12y ago
      Not necessarily. Some Atoms (e.g. Centerton) are now shipping with ECC support; similarly AMD Kabini processors support ECC.
  - ryao 12y ago
    The "ECC issue" is relevant for all software and by extension, all filesystems. ZFS eliminates so many problems in the storage stack that things it cannot eliminate (like bit-flips from not having ECC) become more noticeable by virtue of fewer things going wrong. Here is some documentation on this: http://open-zfs.org/wiki/Hardware#ECC_Memory http://open-zfs.org/wiki/Hardware#ECC_Memory As for ECC in desktops, I am using whatever interactions that I have with vendors to push for it. Here is a public one: https://dellideas.secure.force.com/idea2ReadIdea?v=1403803188582&Id=087700000009EvYAAU https://dellideas.secure.force.com/idea2ReadIdea?v=140380318...
  - rbanffy 12y ago
    > I'd love to run ECC RAM, but I'd have to buy a much more expensive processor, a much more expensive mainboard, and all of my RAM would have to be swapped for moderately more expensive replacements. And DDR3+ is not cheap. It's cheaper than losing your data. We spend 20% more on drives than we have disk space and we do it because we love our data. In most use cases, it's wise to trade some performance (in the form of a slower, single socket, less memory) setup for reliability. Unless you need all the performance you can possibly get, a slightly slower but more reliable machine is sound investment. > if ECC consists of a ninth parity bit A single bit will allow you to detect an error in a given unit, but not correct it. For correcting a single-bit error, you'd need to pinpoint which bit got flipped and that would require a second parity bit.
  - mng2 12y ago
    I went ECC a couple months ago and managed to find some parts that didn't break the bank, mostly because I held back and stayed a generation behind: GA-6UASL3, E3-1230V2. There are other sacrifices: the motherboard is rather no-frills, and the memory bandwidth isn't that great. Besides price, another issue against server hardware is that it's generally designed for a server case and therefore doesn't play well with enthusiast cases. The board I got is a little odd in that it uses a layout more typical of consumer boards, though the socket placement is still not ideal.

mrb 12y ago

This is not the reason why people recommend ECC. ZFS is so good at detecting corruption (CKSUM errors) that when it happens it is hard to tell what caused it: faulty RAM corrupting data before being written to disk or after being read from disk, or data corrupted on disk? ECC simply helps reduce a good chunk of corruption errors caused by faulty RAM. That said, in my experience there are a few tell-tale signs that RAM is the issue: when you see CKSUM errors poping up infrequently, and not being attributed to consistently the same drive(s).

byuu 12y ago
I haven't had any issues so far using four drives mirrored on ZFS, but the ECC issue certainly worries me. I'd love to run ECC RAM, but I'd have to buy a much more expensive processor, a much more expensive mainboard, and all of my RAM would have to be swapped for moderately more expensive replacements. And DDR3+ is not cheap. I'm curious though, if ECC consists of a ninth parity bit, is there any reason why a memory controller can't be designed that would, worst case, use every other (identical) stick just to get parity bit(s), as a BIOS configurable option? Sure it'd halve your RAM, or you'd pay a lot more in RAM costs, but not having to buy Xeon processors and mainboards, and getting to reuse your existing RAM, would be worth buying an extra stick of RAM in my opinion. It seems as processes keep getting smaller, and RAM sizes keep getting larger, that the effects of cosmic radiation are just going to keep getting worse. If we can't get desktop CPUs and mainboards to just switch to ECC, surely this would at least be a better-than-nothing option.
- _ihaque 12y ago
  Most modern CPUs have the memory controller integrated on-die, so you'd need a different class of CPU -- and then you're right back to buying a Xeon.
  - rodgerd 12y ago
    Not necessarily. Some Atoms (e.g. Centerton) are now shipping with ECC support; similarly AMD Kabini processors support ECC.
- ryao 12y ago
  The "ECC issue" is relevant for all software and by extension, all filesystems. ZFS eliminates so many problems in the storage stack that things it cannot eliminate (like bit-flips from not having ECC) become more noticeable by virtue of fewer things going wrong. Here is some documentation on this: http://open-zfs.org/wiki/Hardware#ECC_Memory http://open-zfs.org/wiki/Hardware#ECC_Memory As for ECC in desktops, I am using whatever interactions that I have with vendors to push for it. Here is a public one: https://dellideas.secure.force.com/idea2ReadIdea?v=1403803188582&Id=087700000009EvYAAU https://dellideas.secure.force.com/idea2ReadIdea?v=140380318...
- rbanffy 12y ago
  > I'd love to run ECC RAM, but I'd have to buy a much more expensive processor, a much more expensive mainboard, and all of my RAM would have to be swapped for moderately more expensive replacements. And DDR3+ is not cheap. It's cheaper than losing your data. We spend 20% more on drives than we have disk space and we do it because we love our data. In most use cases, it's wise to trade some performance (in the form of a slower, single socket, less memory) setup for reliability. Unless you need all the performance you can possibly get, a slightly slower but more reliable machine is sound investment. > if ECC consists of a ninth parity bit A single bit will allow you to detect an error in a given unit, but not correct it. For correcting a single-bit error, you'd need to pinpoint which bit got flipped and that would require a second parity bit.
- mng2 12y ago
  I went ECC a couple months ago and managed to find some parts that didn't break the bank, mostly because I held back and stayed a generation behind: GA-6UASL3, E3-1230V2. There are other sacrifices: the motherboard is rather no-frills, and the memory bandwidth isn't that great. Besides price, another issue against server hardware is that it's generally designed for a server case and therefore doesn't play well with enthusiast cases. The board I got is a little odd in that it uses a layout more typical of consumer boards, though the socket placement is still not ideal.