6 ms·

> What's the usecase when it can be beneficial to have expensive ECC RAM instead of the ordinary one? Anyone using ZFS will (or should!) care about ECC support

> What's the usecase when it can be beneficial to have expensive ECC RAM instead of the ordinary one? Anyone using ZFS will (or should!) care about ECC support. [0] Lots of people build their own NAS/SAN boxes, so ECC support on a desktop CPU at a reasonable price point would be very appreciated. Currently you need to buy specific model CPUs (Celeron or Xeon, IIRC) to get ECC support from Intel. [1] [0] https://serverfault.com/questions/454736/non-ecc-memory-with-zfs-a-stupid-idea https://serverfault.com/questions/454736/non-ecc-memory-with... [1] https://ark.intel.com/search/advanced?ECCMemory=true&MarketSegment=DT https://ark.intel.com/search/advanced?ECCMemory=true&MarketS...

snuxoll 10y ago
This is why I ultimately ended up buying an HP ML10 for my FreeNAS box, $300 for a Xeon E3-1220, board and iLO ended up being cheaper than buying a barebones + the CPU (which is just shy of $200 retail). I would like to have something that supports more memory, but to get anything with more sockets or support for larger DIMM's you start looking at the Broadwell-E or Xeon-E5 chips and those are considerably more expensive (my ThinkServer TD340 cost me $700 as an open box with a E5-2403v2 and 8GB of RAM installed). If Zen client chips have full ECC support and can handle 64GB of memory I'll be sold easily on an upgrade for my TrueNAS box, then I'll anxiously await some lower-cost (4/8c) dual-socket server CPU's and swap out my TD340 with some supermicro barebones build.
- lultimouomo 10y ago
  Do you know how much power does it use? If so, could you add some details (number of HDDs, typical load - light, medium, heavy)? I'm looking into building a home server and would like something decently beefy, not too power hungry and possibly with ECC.
  - snuxoll 10y ago
    I know the ML10 only has a 300W PSU, but I haven't bothered to take a kill-a-watt as the ML10 itself would be outshined by the rest of my lab gear. The E3-1220 is usually idle and frequency scaled back unless something like a ZFS scrub is running, I've got ~45W of PCIe cards (SAS HBA, 10GBe NIC and 4x1GBe NIC), 4x8GB sticks of DDR3 UDIMM's probably uses 12W. I'd say without drives it probably draws no more than 150W on average, unless I'm doing something CPU intensive. My drives all sit in an external SAS enclosure, since I wanted more than 4 drive bays that the LFF expansion bracket provided. Anyway, the ML10 is pretty decent for light-medium work. It's got 4 really fast cores and 32GB of RAM is adequate for most home server use (this was why I got the TD340 though, I've got 72GB in it) - just beware that the Gen1 units don't include the drive bracket so if you want more than a single HDD you have to purchase one or get an external SAS enclosure. EDIT: the ML10 is basically silent too, even under load - I can't hear the fans unless I try (though my SAS enclosure makes up for this by being the loudest bit of kit in my lab).
solarengineer 10y ago
To be clear: ZFS performs a checksum _before_ the data is written to disk. This checksum proves to be highly useful to ZFS when checking whether the data being read back is correct or not. However, the checksum and the data is also only as good as the memory in which it is stored during computation. Since ZFS takes care of everything else other than memory, you'd use ECC if the data being stored is important enough that you want to ensure that ZFS gets healthy checksums.
- zer0t3ch 10y ago
  If ZFS is anything like BTRFS, then I don't think ECC is super necessary. Yeah, it'll prevent problem from eventually cropping up when the RAM shits itself, but it's not like those issues compound silently in the night. When my ram started going a bit down the drain, (Flipping the occasional bit. Rare enough that the system still ran fine) I was hounded with checksum errors and any attempt to work with the data was met with error messages until I got new RAM. I don't think ECC is really all that important unless you're dealing with data worth thousands of dollars or more.
  - guitarbill 10y ago
    IDK, I've seen some odd RAM failure modes where the base OS would run fine, but then depending on the load (e.g. games) some weird stuff would happen like a BSOD after exactly <x> minutes of gameplay. Once ZFS writes corrupted metadata with corrupted checksums to the disk, it's very hard to recover that data. Yes, ZFS is not backup, but the implications of serving clients messed up data is also worrying in certain business cases. Given that ECC RAM isn't (much?) more expensive, why not just use ECC? (For my microserver I remember it being the same price as non-ECC.) Well, because CPUs with ECC support are rare, etc.
    - boomboomsubban 10y ago
      The RAM itself isn't much more expensive, it's a compatible motherboard and CPU that add to the price. If you can't afford even a temporary loss of data it's well worth the cost. That is true with whatever filesystem you choose, and ZFS is always a great choice.
weitzj 10y ago
There is nothing special about using ZFS with/without ECC. You are getting the same benefits using any filesystem while running with ECC. Also the "myth" that wrong checksum calculations due to a bitflip will degrade a ZFS filesystem even faster is not valid. There are some articles/newsgroups around explaining this in much more detail. So: Using ZFS on non ECC is not inherently unsafer than other filesystems on non ECC.
- iso-8859-1 10y ago
  And here is Matt Ahrens, a ZFS co-founder saying the same ("there's nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem"): https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=26303271#p26303271 https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=...
seanp2k2 10y ago
Great use-case for the server-grade Atoms. I have a freenas build with an Avoton 8-core and it's fantastic. Asrock Rack c2750D4i, check it out. Can't wait for the new Atom server chips as well, and I hope AMD launches a competitor in the low-power / storage server market.
- lultimouomo 10y ago
  I am looking into replacing my home server / NAS, and I considered the atom server chips, but they seem just so expensive for their computing power, and also not very power efficient. The 8 core C2750 has a passmark of 3800[0], with an abysmal single thread performance of 579, and still has a TDP of 20W, and the mobo+cpu assemblies go for ~400$. For way less, if you're willing to five up on ECC, you can get an i5-7500T and a pretty good motherboard, bringing your TDP to 35W but having a passmark of 7055, and most importantly, a single thread rating of 1924. This might not matter much if you're use is exclusively NAS, but I will probably end up running some virtualized or containerized server, or streaming video possibly transcoding on the fly, and I'm afraid the Atom might become a bottleneck. Is there a solution that is somewhat competitive with the i3/5/7 on price and power, and has ECC? And, ideally, that comes in a Mini-ITX form factor? [0] Obviously the PassMark score is only a ballpark estimate to get an idea of how fast is a chip, but still.
  - yuhong 10y ago
    Look at Xeon E3, for example: https://www.newegg.com/Product/Product.aspx?Item=N82E16819117611 https://www.newegg.com/Product/Product.aspx?Item=N82E1681911...
    - lultimouomo 10y ago
      Thanks. It looks there are reasonably priced mini-ITX boards that support Skylake Xeons (a note to this that are considering this: v5 E3 (Skylake) use the same socket as Cores, but need a different chipset). The 80W TDP worries me about though, cooling, noise and money/pollution wise (the extra 45W make around 400kWh/year). Considering the CPU will be idle most of the time, do you have figures about the actual idle consumption of an E3 based machine?

snuxoll 10y ago

This is why I ultimately ended up buying an HP ML10 for my FreeNAS box, $300 for a Xeon E3-1220, board and iLO ended up being cheaper than buying a barebones + the CPU (which is just shy of $200 retail). I would like to have something that supports more memory, but to get anything with more sockets or support for larger DIMM's you start looking at the Broadwell-E or Xeon-E5 chips and those are considerably more expensive (my ThinkServer TD340 cost me $700 as an open box with a E5-2403v2 and 8GB of RAM installed). If Zen client chips have full ECC support and can handle 64GB of memory I'll be sold easily on an upgrade for my TrueNAS box, then I'll anxiously await some lower-cost (4/8c) dual-socket server CPU's and swap out my TD340 with some supermicro barebones build.

lultimouomo 10y ago
Do you know how much power does it use? If so, could you add some details (number of HDDs, typical load - light, medium, heavy)? I'm looking into building a home server and would like something decently beefy, not too power hungry and possibly with ECC.
- snuxoll 10y ago
  I know the ML10 only has a 300W PSU, but I haven't bothered to take a kill-a-watt as the ML10 itself would be outshined by the rest of my lab gear. The E3-1220 is usually idle and frequency scaled back unless something like a ZFS scrub is running, I've got ~45W of PCIe cards (SAS HBA, 10GBe NIC and 4x1GBe NIC), 4x8GB sticks of DDR3 UDIMM's probably uses 12W. I'd say without drives it probably draws no more than 150W on average, unless I'm doing something CPU intensive. My drives all sit in an external SAS enclosure, since I wanted more than 4 drive bays that the LFF expansion bracket provided. Anyway, the ML10 is pretty decent for light-medium work. It's got 4 really fast cores and 32GB of RAM is adequate for most home server use (this was why I got the TD340 though, I've got 72GB in it) - just beware that the Gen1 units don't include the drive bracket so if you want more than a single HDD you have to purchase one or get an external SAS enclosure. EDIT: the ML10 is basically silent too, even under load - I can't hear the fans unless I try (though my SAS enclosure makes up for this by being the loudest bit of kit in my lab).

solarengineer 10y ago

To be clear: ZFS performs a checksum _before_ the data is written to disk. This checksum proves to be highly useful to ZFS when checking whether the data being read back is correct or not. However, the checksum and the data is also only as good as the memory in which it is stored during computation. Since ZFS takes care of everything else other than memory, you'd use ECC if the data being stored is important enough that you want to ensure that ZFS gets healthy checksums.

zer0t3ch 10y ago
If ZFS is anything like BTRFS, then I don't think ECC is super necessary. Yeah, it'll prevent problem from eventually cropping up when the RAM shits itself, but it's not like those issues compound silently in the night. When my ram started going a bit down the drain, (Flipping the occasional bit. Rare enough that the system still ran fine) I was hounded with checksum errors and any attempt to work with the data was met with error messages until I got new RAM. I don't think ECC is really all that important unless you're dealing with data worth thousands of dollars or more.
- guitarbill 10y ago
  IDK, I've seen some odd RAM failure modes where the base OS would run fine, but then depending on the load (e.g. games) some weird stuff would happen like a BSOD after exactly <x> minutes of gameplay. Once ZFS writes corrupted metadata with corrupted checksums to the disk, it's very hard to recover that data. Yes, ZFS is not backup, but the implications of serving clients messed up data is also worrying in certain business cases. Given that ECC RAM isn't (much?) more expensive, why not just use ECC? (For my microserver I remember it being the same price as non-ECC.) Well, because CPUs with ECC support are rare, etc.
  - boomboomsubban 10y ago
    The RAM itself isn't much more expensive, it's a compatible motherboard and CPU that add to the price. If you can't afford even a temporary loss of data it's well worth the cost. That is true with whatever filesystem you choose, and ZFS is always a great choice.

weitzj 10y ago

There is nothing special about using ZFS with/without ECC. You are getting the same benefits using any filesystem while running with ECC. Also the "myth" that wrong checksum calculations due to a bitflip will degrade a ZFS filesystem even faster is not valid. There are some articles/newsgroups around explaining this in much more detail. So: Using ZFS on non ECC is not inherently unsafer than other filesystems on non ECC.

iso-8859-1 10y ago
And here is Matt Ahrens, a ZFS co-founder saying the same ("there's nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem"): https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=26303271#p26303271 https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=...

seanp2k2 10y ago

Great use-case for the server-grade Atoms. I have a freenas build with an Avoton 8-core and it's fantastic. Asrock Rack c2750D4i, check it out. Can't wait for the new Atom server chips as well, and I hope AMD launches a competitor in the low-power / storage server market.

lultimouomo 10y ago
I am looking into replacing my home server / NAS, and I considered the atom server chips, but they seem just so expensive for their computing power, and also not very power efficient. The 8 core C2750 has a passmark of 3800[0], with an abysmal single thread performance of 579, and still has a TDP of 20W, and the mobo+cpu assemblies go for ~400$. For way less, if you're willing to five up on ECC, you can get an i5-7500T and a pretty good motherboard, bringing your TDP to 35W but having a passmark of 7055, and most importantly, a single thread rating of 1924. This might not matter much if you're use is exclusively NAS, but I will probably end up running some virtualized or containerized server, or streaming video possibly transcoding on the fly, and I'm afraid the Atom might become a bottleneck. Is there a solution that is somewhat competitive with the i3/5/7 on price and power, and has ECC? And, ideally, that comes in a Mini-ITX form factor? [0] Obviously the PassMark score is only a ballpark estimate to get an idea of how fast is a chip, but still.
- yuhong 10y ago
  Look at Xeon E3, for example: https://www.newegg.com/Product/Product.aspx?Item=N82E16819117611 https://www.newegg.com/Product/Product.aspx?Item=N82E1681911...
  - lultimouomo 10y ago
    Thanks. It looks there are reasonably priced mini-ITX boards that support Skylake Xeons (a note to this that are considering this: v5 E3 (Skylake) use the same socket as Cores, but need a different chipset). The 80W TDP worries me about though, cooling, noise and money/pollution wise (the extra 45W make around 400kWh/year). Considering the CPU will be idle most of the time, do you have figures about the actual idle consumption of an E3 based machine?