4 ms·

> Examples of things you don't want to do if you want to keep your data intact include using non-ECC RAM I'm planning to build a (Linux) development workstatio

by mhw 12y ago

> Examples of things you don't want to do if you want to keep your data intact include using non-ECC RAM I'm planning to build a (Linux) development workstation soon and was intending to run ZFS on at least a proportion of the disks in the system, if not the whole thing. But building a workstation that supports ECC appears to be ridiculously hard and expensive - most of the Haswell CPUs don't support ECC. Can anyone point me at a sensible CPU/motherboard combo that's comparable to something like an i7-4770k/Gigabyte GA-Z87MX-D3H? (As an indication of the state of the component market, pcpartpicker.com doesn't support searching CPUs or motherboards by ECC support, but it does allow you to search by what colour the motherboard is!)

rockdoe 12y ago
You need a Xeon v3 for the ECC support, basically. Also obviously getting ECC ram for stability and getting a K series CPU for overclocking is just an incompatible mindset.
- moletown 12y ago
  You can use ECC on some i3 CPUs [1] as long as the motherboard supports it. Supermicro has a X10 micro-atx board [2] which would work. [1] http://ark.intel.com/search/advanced?s=t&FamilyText=4th%20Generation%20Intel%C2%AE%20Core%E2%84%A2%20i3%20Processors&MarketSegment=DT&ECCMemory=true http://ark.intel.com/search/advanced?s=t&FamilyText=4th%20Ge... [2] http://www.supermicro.com/products/motherboard/Xeon/C220/X10SLM-F.cfm http://www.supermicro.com/products/motherboard/Xeon/C220/X10...
  - voltagex_ 12y ago
    I wonder if an i3 has enough grunt for ZFS. I definitely notice the difference between a middling i3 and my i7 2600k.
  - voltagex_ 12y ago
    Fiddling with the filters on that page, it seems some Atom processors support ECC, too - http://ark.intel.com/products/81270/Intel-Atom-Processor-C2508-2M-Cache-1_25-GHz http://ark.intel.com/products/81270/Intel-Atom-Processor-C25...
Xylakant 12y ago
While the recommendation is in general sound, I don't think the article applies to using ZFS on a workstation. It's about using ZFS as a storage subsystem (of multiple TB size) where data must be kept safe and without errors. You can still get most of the benefits on a workstation with non-ECC RAM, consider that any other file system would have the same challenge to deal with on non-ECC RAM. So, as in all things, it's a money/benefits tradeoff: Does the data you're working with warrant spending a ridiculous amount of money?
Nanzikambe 12y ago
YMMV, but I've used ZFS on all my workstations and servers for over 2 years now. Many of them are ZFS on top of LUKS/cryptsetup for FDE. None of the machines use ECC RAM, and due lots of things (ie. me doing dumb shit, bad cutting edge kernel builds, fubar GFX drivers causing X to boot to black screen etc) I frequently either hard reset them, or Alt + Prtscreen + REISUB them, I've never had a single issue with ZFS. By comparison, I've had issues with virtually every other filesystem that exists for Linux: ext(2,3,tho not 4), reiserfs, and especially the much hated xfs, death to xfs etc :) I run his hardware, picked because... well cheap: GIGABYTE GA-990FXA-UD5 ( AMD 990FX/SB950 - Socket AM3+ - FSB5200 ) 4 x DDR3 4GB DDR1866 (PC3-15000) - KINGSTON HyperX [KHX1866C9D3/4G] or 4 x 16gb of the same type as above AMD FX 8-Core FX-8320 [Socket AM3+ - 1000Kb - 3.5 GHz - 32nm ] The ZFS all include root + swap running on ZFS with FDE, this sort of thing for workstations: NAME USED AVAIL REFER MOUNTPOINT rpool 13.2G 25.9G 30K none rpool/root 11.1G 25.9G 5.10G / rpool/root/home-users 6.02G 25.9G 6.02G /home rpool/swap 2.06G 27.4G 646M - servers: NAME USED AVAIL REFER MOUNTPOINT rpool0/ROOT/ROOT 6G 1.42T 198K / rpool0/ROOT/swap 32G 1.42T 700M - rpool0/ROOT/BACKUP 676G 1.42T 676G /nfs/backup rpool0/ROOT/HOME 14.1G 1.42T 14.1G /home rpool0/ROOT/MUSIC 358G 1.42T 358G /nfs/music rpool0/ROOT/portage 14.9G 1.42T 14.9G /nfs/portage rpool0/ROOT/vmware 45.9G 1.42T 45.9G /vmware The server hosts several virtual machines, many of which have their own root over NFS, the point being that the ZFS gets plenty of use.
- dTal 12y ago
  You ever try JFS? I standardised on it for all my machines a few years ago and it's served me very well - fast fsck and no corruption so far. Bit long in the tooth now but that's no bad thing for a filesystem.
  - Nanzikambe 12y ago
    I have not, but the only reason for not trying it is because of my increasing use of RAID in hardware and VMs, I wanted something that was completely agnostic to disks, which is something ZFS delivers but JFS didn't seem to when I checked. Basically I wanted something that doesn't intertwine the concept of file-system with the concept of disks or partitions, but as space. So changes to the underlying pool of disks (or partitions, or loopback files, etc) don't mean so much painful rebuilding for me, something I spent many hundreds of hours on in the past, often at stupid o'clock in the morning. Did I mention how much I hate XFS? That's why :)
skorgu 12y ago
Most AMD chips support ECC on the cpu, many compatible motherboards include it but don't bother to specify on the spec sheet. I was running an Asus M5A97 with ECC with ZFS for a few before it died a tragic early death.
e12e 12y ago
I feel your pain -- I've been trying to figure out the state of (cheap-ish) ecc support for a while now (the freenas forums can be a big help here). Basically you can roll the dice with AMD, or just go with Intel (what I'll be doing). If you want reasonable performance, get a socket 1150 xeon v3 (basically a re-branded i7 with ecc support). The low-mid xeon v3 appear to score pretty well on a price/performance ratio. Can't be overclocked, and you'd probably get more single thread performance/price with an i5 -- but apart from that they seem pretty solid. As others have mentioned, (some) i3s are also an option -- but as you'll likely have to pay a little more for a main board with ecc support, that doesn't make much sense as I see it. The final option is to get one of the newish atom based Avoton boards, like ASROCK C2750D4 INTEL C2750 AVOTON OCTACORE MITX -- not for a workstation, though. But should be nice as a NAS. If anyone knows of a useful overview of AMD cpus and mainboard combinations that support ECC -- please let us know. I've yet to find anything beyond the anecdotal "Here's my build, and it's got ECC sticks in it, and maybe the ECC is actually enabled, but I haven't really checked."
vardump 12y ago
Supermicro has a ton of mainboards with ECC support. Many of them have integrated "RAID" or JBOD. 8 SAS + 6 SATA is typical. I haven't checked the prices for a while, but last time I bought, it was about $250 for a mainboard with integrated 8x SAS JBOD and 6x SATA. Very cheap, when you consider no separate JBOD card is required. Those boards and their SAS chipsets (LSI SAS2008, etc.) tend to be well supported on platforms capable of running ZFS, like Solaris, FreeBSD etc. Other typical goodies include IPMI (integrated IP-KVM), good quality dual NICs and generally good quality components.
jeffmk 12y ago
I don't know what your budget is, but I found that the ASRock E3C226D2I Mini ITX "Server Motherboard" fit my needs well. Being an LGA 1150 board, it seems to support the i7-4770k CPU you mentioned and is about $100 more than your comparable setup. It has 2 USB 3.0 ports, supports up to 16GiB of RAM, and has 6 SATA 6Gb/s ports, so I think it would also work well for a workstation board, if you added a video/audio card -- HDMI out is the only glaring omission for a workstation. I'm using it in my FreeNAS fileserver and have been happy with it.

by mhw 12y ago

rockdoe 12y ago

You need a Xeon v3 for the ECC support, basically. Also obviously getting ECC ram for stability and getting a K series CPU for overclocking is just an incompatible mindset.

moletown 12y ago
You can use ECC on some i3 CPUs [1] as long as the motherboard supports it. Supermicro has a X10 micro-atx board [2] which would work. [1] http://ark.intel.com/search/advanced?s=t&FamilyText=4th%20Generation%20Intel%C2%AE%20Core%E2%84%A2%20i3%20Processors&MarketSegment=DT&ECCMemory=true http://ark.intel.com/search/advanced?s=t&FamilyText=4th%20Ge... [2] http://www.supermicro.com/products/motherboard/Xeon/C220/X10SLM-F.cfm http://www.supermicro.com/products/motherboard/Xeon/C220/X10...
- voltagex_ 12y ago
  I wonder if an i3 has enough grunt for ZFS. I definitely notice the difference between a middling i3 and my i7 2600k.
- voltagex_ 12y ago
  Fiddling with the filters on that page, it seems some Atom processors support ECC, too - http://ark.intel.com/products/81270/Intel-Atom-Processor-C2508-2M-Cache-1_25-GHz http://ark.intel.com/products/81270/Intel-Atom-Processor-C25...

Xylakant 12y ago

While the recommendation is in general sound, I don't think the article applies to using ZFS on a workstation. It's about using ZFS as a storage subsystem (of multiple TB size) where data must be kept safe and without errors. You can still get most of the benefits on a workstation with non-ECC RAM, consider that any other file system would have the same challenge to deal with on non-ECC RAM. So, as in all things, it's a money/benefits tradeoff: Does the data you're working with warrant spending a ridiculous amount of money?

Nanzikambe 12y ago

YMMV, but I've used ZFS on all my workstations and servers for over 2 years now. Many of them are ZFS on top of LUKS/cryptsetup for FDE. None of the machines use ECC RAM, and due lots of things (ie. me doing dumb shit, bad cutting edge kernel builds, fubar GFX drivers causing X to boot to black screen etc) I frequently either hard reset them, or Alt + Prtscreen + REISUB them, I've never had a single issue with ZFS. By comparison, I've had issues with virtually every other filesystem that exists for Linux: ext(2,3,tho not 4), reiserfs, and especially the much hated xfs, death to xfs etc :) I run his hardware, picked because... well cheap: GIGABYTE GA-990FXA-UD5 ( AMD 990FX/SB950 - Socket AM3+ - FSB5200 ) 4 x DDR3 4GB DDR1866 (PC3-15000) - KINGSTON HyperX [KHX1866C9D3/4G] or 4 x 16gb of the same type as above AMD FX 8-Core FX-8320 [Socket AM3+ - 1000Kb - 3.5 GHz - 32nm ] The ZFS all include root + swap running on ZFS with FDE, this sort of thing for workstations: NAME USED AVAIL REFER MOUNTPOINT rpool 13.2G 25.9G 30K none rpool/root 11.1G 25.9G 5.10G / rpool/root/home-users 6.02G 25.9G 6.02G /home rpool/swap 2.06G 27.4G 646M - servers: NAME USED AVAIL REFER MOUNTPOINT rpool0/ROOT/ROOT 6G 1.42T 198K / rpool0/ROOT/swap 32G 1.42T 700M - rpool0/ROOT/BACKUP 676G 1.42T 676G /nfs/backup rpool0/ROOT/HOME 14.1G 1.42T 14.1G /home rpool0/ROOT/MUSIC 358G 1.42T 358G /nfs/music rpool0/ROOT/portage 14.9G 1.42T 14.9G /nfs/portage rpool0/ROOT/vmware 45.9G 1.42T 45.9G /vmware The server hosts several virtual machines, many of which have their own root over NFS, the point being that the ZFS gets plenty of use.

dTal 12y ago
You ever try JFS? I standardised on it for all my machines a few years ago and it's served me very well - fast fsck and no corruption so far. Bit long in the tooth now but that's no bad thing for a filesystem.
- Nanzikambe 12y ago
  I have not, but the only reason for not trying it is because of my increasing use of RAID in hardware and VMs, I wanted something that was completely agnostic to disks, which is something ZFS delivers but JFS didn't seem to when I checked. Basically I wanted something that doesn't intertwine the concept of file-system with the concept of disks or partitions, but as space. So changes to the underlying pool of disks (or partitions, or loopback files, etc) don't mean so much painful rebuilding for me, something I spent many hundreds of hours on in the past, often at stupid o'clock in the morning. Did I mention how much I hate XFS? That's why :)

skorgu 12y ago

Most AMD chips support ECC on the cpu, many compatible motherboards include it but don't bother to specify on the spec sheet. I was running an Asus M5A97 with ECC with ZFS for a few before it died a tragic early death.

e12e 12y ago

I feel your pain -- I've been trying to figure out the state of (cheap-ish) ecc support for a while now (the freenas forums can be a big help here). Basically you can roll the dice with AMD, or just go with Intel (what I'll be doing). If you want reasonable performance, get a socket 1150 xeon v3 (basically a re-branded i7 with ecc support). The low-mid xeon v3 appear to score pretty well on a price/performance ratio. Can't be overclocked, and you'd probably get more single thread performance/price with an i5 -- but apart from that they seem pretty solid. As others have mentioned, (some) i3s are also an option -- but as you'll likely have to pay a little more for a main board with ecc support, that doesn't make much sense as I see it. The final option is to get one of the newish atom based Avoton boards, like ASROCK C2750D4 INTEL C2750 AVOTON OCTACORE MITX -- not for a workstation, though. But should be nice as a NAS. If anyone knows of a useful overview of AMD cpus and mainboard combinations that support ECC -- please let us know. I've yet to find anything beyond the anecdotal "Here's my build, and it's got ECC sticks in it, and maybe the ECC is actually enabled, but I haven't really checked."

vardump 12y ago

Supermicro has a ton of mainboards with ECC support. Many of them have integrated "RAID" or JBOD. 8 SAS + 6 SATA is typical. I haven't checked the prices for a while, but last time I bought, it was about $250 for a mainboard with integrated 8x SAS JBOD and 6x SATA. Very cheap, when you consider no separate JBOD card is required. Those boards and their SAS chipsets (LSI SAS2008, etc.) tend to be well supported on platforms capable of running ZFS, like Solaris, FreeBSD etc. Other typical goodies include IPMI (integrated IP-KVM), good quality dual NICs and generally good quality components.

jeffmk 12y ago

I don't know what your budget is, but I found that the ASRock E3C226D2I Mini ITX "Server Motherboard" fit my needs well. Being an LGA 1150 board, it seems to support the i7-4770k CPU you mentioned and is about $100 more than your comparable setup. It has 2 USB 3.0 ports, supports up to 16GiB of RAM, and has 6 SATA 6Gb/s ports, so I think it would also work well for a workstation board, if you added a video/audio card -- HDMI out is the only glaring omission for a workstation. I'm using it in my FreeNAS fileserver and have been happy with it.