7 ms·

With ZFS allowing for very robust data integrity on disk, the weak point in most budget NAS setups these days is going to be memory errors. It’s unfortunate tha

by wyager 5y ago

With ZFS allowing for very robust data integrity on disk, the weak point in most budget NAS setups these days is going to be memory errors. It’s unfortunate that there are so few low-end options with any kind of ECC. The PCEngines APU2 has an ECC option, but it does not have the appropriate I/O for redundant storage.

3np 5y ago
If you're doing ZFS where that I/O bandwidth is a limiting factor, I'm thinking the 4GB of the APU2 makes it not interesting for ZFS in the first place (depending on the workloads of course, but you're probably going to want a bigger ARC). I can see it work well as a sink for automated replicated snapshots, where I/O performance is not that critical.
- dsr_ 5y ago
  Yup. Use mdadm to do RAID5, 6 or 10 and let it sit in your friend's place as a target for encrypted snapshots; do the same in exchange and you both have reliable offsite backups. If you need a big restore, go get the machine physically and bring it back to your network for full gigabit speeds.
- blibble 5y ago
  unless you have several hundred TB, 4GB should be fine it's quite happy with a 40tb array I have on a 2010 server with 4GB of ram (and I've run ZFS on every laptop I've owned in the last decade or so)
- wyager 5y ago
  ZFS works fine these days in low-RAM environments. As you say, you won’t benefit much from ARC, but for most NAS workloads that is fine.
geerlingguy 5y ago
The two problems I see are ZFS really wants a bit of a beefier machine, plus all the RAM you can throw at it. That pretty much excludes anything low end by default (not to mention more expensive ECC memory).
- handrous 5y ago
  I think ZFS only needs lots of memory if you have certain features (notably, deduplication) enabled. Meanwhile, you're still a lot better off, data-integrity-wise, with ZFS and non-ECC memory than you are with most other filesystems with non-ECC memory.
  - nine_k 5y ago
    XFS also does checksums, but only for metadata.
  - 3np 5y ago
    It depends on the workload. With insufficent memory and/or latency, ZFS can perform orders of magnitude worse than ext4/xfs, even when tuned. (I speak from experience) Not exactly the same, but did you ever try ZFS on a commodity USB stick? It can get really bad when circumstances are not in favor. With not too much concurrency it can perform quite well even under constraints, though!
- KaiserPro 5y ago
  I've run ZFS on 2/4/8 & 16 gig machines. Yes, more ram makes things a bit faster, as you get a bigger arc cache. however as you point out, 4gigs is more than enough to surface a 1gig connection. Especially on a bunch of SSDs
exhilaration 5y ago
What's the solution for a budget NAS setup with ECC? Stick with Intel/AMD (for now)?
nine_k 5y ago
For that reason, my choice for a home NAS system board was a retired itx server board with a 20W Atom, 8 SATA ports, and ECC memory. Comparable to RPi at compute power, but with vastly better I/O and, again, ECC RAM, because I don't want my data to have a chance to be silently corrupted. I think there can be a niche for a system board like that, only open and based on ARM. IDK if any smaller ARM processors support ECC, though.
- bombcar 5y ago
  Do you have more details on that - part number or etc? I have an older beast of a machine that I'm thinking of replacing to get power consumption down, but I'd like to keep ECC for ZFS.
  - nine_k 5y ago
    I just trawled eBay for "itx c2750". 2550 can also be considered for lower power, or 3750 for higher price and more capacity. Cxx58 parts can be considered instead of Cxx50 parts. Of course all these CPUs are very old, 7-9 years since introduction. But it's ok by me, they are reasonably cold, have 4 or 8 cores, and I'm looking for a cold, silent, and reliable system, not a screaming fast system. At worst, a 10GbE card can be installed into the lone PCIe slot, but I don't expect my rig to saturate even the 1GbE built-in interface. (My board has four of these anyway.) As of motherboard producers, SuperMicro makes the right kind: passive cooling, most / all ports wired, internal USB and / or m.2. Gigabyte made more consumer-oriented variants, but also serviceable.
dpedu 5y ago
ZFS doesn't need ECC ram to work correctly. This is an incorrect but commonly repeated myth. See: https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-y...
- ed25519FUUU 5y ago
  But wouldn’t it work better with ECC?
- segfaultbuserr 5y ago
  The original comment did not claim that ZFS specifically needs ECC to run correctly. Quote the OP, > With ZFS allowing for very robust data integrity on disk, the weak point in most budget NAS setups these days is going to be memory errors. You see, it didn't say "no ECC, no ZFS." It merely says: if your goal is to eliminate data corruption, once you've removed filesystem errors from the equation by using ZFS, the next term to eliminate is memory errors - which is entirely correct.
- wyager 5y ago
  It doesn’t “need” it, but ZFS + ECC is a lot more effective than ZFS + high-BER RAM. Once you fix disk corruption, the next thing you’ll probably worry about is memory corruption.
logifail 5y ago
> the weak point in most budget NAS setups these days is going to be memory errors I use a HPE ProLiant MicroServer Gen8 as a (very) budget home/home-office NAS, I'm really not convinced that memory errors are its weak point(!) It has the default (horrible) Celeron G1610T processor, was upgraded on delivery to 16G of memory, but has 60TB of storage - four internal SATA drives plus additional external USB drives. I've tried and so far failed to get it to run stable 10GbE networking. Just sourced a new (old) ConnectX-3 card from eBay to try again...
- gothroach 5y ago
  I use an i3-4370 with 32G of ECC in a Supermicro 4u case as my ZFS NAS - I got it new for the purpose and it's still running strong seven years later. The only changes I've made have been doubling the RAM from 16G, adding in new storage as needed, and adding a dual 10gbit card when I upgraded my network. I tried a couple of 10gbit cards, but ended up getting a half dozen used Intel X520-DA2 cards for dirt cheap. I've not had any issue with these, and my NAS can easily saturate at least one link. The cards have been reliable in every machine I've tried and can be found for $75-100 - not too bad for dual port 10gbit! As a side note, I was surprised to find that i3 supports ECC.
- forty 5y ago
  I use a MicroServer too (an older one). They have ECC memory, so you should indeed be fine on this point. But price and electricity consumption are much higher than a RPi. I agree that the CPU is the weak point (mine has a turion n40l which is slow as hell).
  - logifail 5y ago
    > mine has a turion n40l Heh, I have two of those older models lurking in the garage from some older experimental stuff. They were at one point configured as a DRBD two-node cluster, which was fun, although performance was dire.
blibble 5y ago
I've run ZFS on my personal (read: cheap) hardware for several hundred machine years and never had a problem with standard RAM (and issues would be detectable with the monthly scrub) obviously ECC is better but you're no worse off than you would be with mdraid or some awful hardware controller
- wyager 5y ago
  > and issues would be detectable with the monthly scrub Depends where the corruption happened. Scrub can’t detect data that got corrupted before the checksum was taken.
denimnerd42 5y ago
A ryzen system with ecc can be had for less than $400 build your own.
- KaiserPro 5y ago
  yes, but even the smallest ryzen sucks down more power and the system is physically bigger. but its way more powerful so can do more stuff.
  - denimnerd42 5y ago
    Yeah low power single board computer systems with all the right features tend to be very expensive and difficult to source without large orders. A ryzen system at 20 or 40 watts idle is something you can probably put up with for $400
1MachineElf 5y ago
I completely agree with you. ZFS has led the way in data integrity and ergonomics, and it's time we got it in a small form factor with reliable RAM. Many commenters have stated that it would be better to go with a Ryzen/ECC or some server-grade Atom board. While that may be true from a performance standpoint, it is not true if your priorities are low power usage and saving space. I have several big beefy servers to run ZFS on my network. What I am missing is something tiny and quiet that can do the same job at a small scale just as well while the big machines are turned off. I am not concerned about maintaining fast I/O or running some RAM-hogging zraid6. I just want 4 separate SSDs running 4 separate spools that I can read/write to anytime and later sync with the big machines when they're online. The even the 4GB RPi would perform that fine - I just don't see a point if the RAM is not ECC.
- dragontamer 5y ago
  > low power usage How about turning the NAS on-and-off as you need it? There's no need to run an "always on" computer unless you want to. If you want to run an IRC bouncer or a webserver, then buy a Rasp. Pi for that. The NAS is primarily storage, which doesn't necessarily need to be always on. (Besides, turning the computer off is the ultimate security move. You can't get that data hacked if the computer isn't even on) ------------ > I have several big beefy servers to run ZFS on my network. What I am missing is something tiny and quiet that can do the same job at a small scale just as well while the big machines are turned off. Ever try just... sticking multiple small-and-cheap hard drives into the clients? Its not too hard to make redundant RAID1-like storage spaces (Windows-clients) or RAID1-like LVM (Linux-clients).
kalleboo 5y ago
> the weak point in most budget NAS setups these days is going to be memory errors In practice, the weak point in most budget NAS setups (1) no backups "because RAID" and then getting hit by a cryptolocker (2) drives dying
- dragontamer 5y ago
  > (2) drives dying Is that really a problem? I'm still able to find 1TB new hard drives at my local microcenter for just $30. So its not really that expensive to set up a 4x drive RAID10 or RAID6 array. 4x 1TB in RAID10 or RAID6 == 2TB with redundancy for $120ish. Go up to 4TB drives ($100 each) and you'll get 8TB capacity for maybe $400ish. Prices suck right now, even for hard drives. (I used to get 6TB hard drives for $150... but the 4TB point still seems reasonable). > (1) no backups and cryptolockers That seems to be easily solved by regular ZFS-snapshots.
  - toast0 5y ago
    Hard drives do die, and while many (possibly most?) deaths are random materials failure, some deaths are correlated to batches and some are firmware problems which are often correlated. Avoiding these is hard. Ideally, you want to get drives from different batches and offset power on times by about a week per drive. Assuming you will notice a failure, obtain a replacement, and replace the failed drive within a week. If that's not realistic, you should wait even longer between drive power on, which makes setup a very long process. It's also hard to ensure drives are from different batches unless you buy in person or get different brands, but there's not four brands to buy from anymore. It's a little easier to do best practices when you replace drives (to get more capacity for example)... Buy one new drive once every other month, and you probably will get different batches, and you'll most likely have sufficient time to replace drives without running into the same firmware bug on all the drives.
  - tinus_hn 5y ago
    If only all these mega corporations that get hit by crypto lockers had heard the magic bullet of ZFS! /s
seltzered_ 5y ago
> The PCEngines APU2 has an ECC option, but it does not have the appropriate I/O from looking at https://pcengines.ch/apu2.htm https://pcengines.ch/apu2.htm , perhaps one could bridge from the miniPCI express to a SAN eSATA tower? (rough example: https://hobo.house/2016/02/03/replacing-failed-linux-raid-disks-with-sans-digital-8bay-esata-tower/ https://hobo.house/2016/02/03/replacing-failed-linux-raid-di... )
colechristensen 5y ago
Ok, I see this a lot. Does anyone have real world experience with meaningful corruption which could be pointed at memory errors? A certain sort of person is very interested in ECC and I have a lot of doubt that it actually causes anybody trouble commensurate with the attention it gets (unless you are Google, and you aren't Google) Even if a few bits are flipped in a media file once during transfer or even permanently... or if the OS state of my storage box gets corrupted and I have to reboot it... so what?
- wyager 5y ago
  RAM BER is shockingly high, like several bits per GB per year. Dozens or hundreds of data corruptions per year does not meet my requirements for even a home NAS, even if most are transient.
  - colechristensen 5y ago
    Well what are your requirements? What are the outcomes you think are going to happen which you are trying to avoid? The "error rate" of, say, a dust mote on my TV screen would seem to be orders of magnitude higher than a few flipped bits per GB*year. I have quite a bit of doubt about how many bit errors would ever hit A) during a time in which I am actually retrieving data (or much much less likely saving) and B) actually hits the surface the data is resident in memory and C) has an impact that raises to a perceptual level.

by wyager 5y ago

3np 5y ago

If you're doing ZFS where that I/O bandwidth is a limiting factor, I'm thinking the 4GB of the APU2 makes it not interesting for ZFS in the first place (depending on the workloads of course, but you're probably going to want a bigger ARC). I can see it work well as a sink for automated replicated snapshots, where I/O performance is not that critical.

dsr_ 5y ago
Yup. Use mdadm to do RAID5, 6 or 10 and let it sit in your friend's place as a target for encrypted snapshots; do the same in exchange and you both have reliable offsite backups. If you need a big restore, go get the machine physically and bring it back to your network for full gigabit speeds.
blibble 5y ago
unless you have several hundred TB, 4GB should be fine it's quite happy with a 40tb array I have on a 2010 server with 4GB of ram (and I've run ZFS on every laptop I've owned in the last decade or so)
wyager 5y ago
ZFS works fine these days in low-RAM environments. As you say, you won’t benefit much from ARC, but for most NAS workloads that is fine.

geerlingguy 5y ago

The two problems I see are ZFS really wants a bit of a beefier machine, plus all the RAM you can throw at it. That pretty much excludes anything low end by default (not to mention more expensive ECC memory).

handrous 5y ago
I think ZFS only needs lots of memory if you have certain features (notably, deduplication) enabled. Meanwhile, you're still a lot better off, data-integrity-wise, with ZFS and non-ECC memory than you are with most other filesystems with non-ECC memory.
- nine_k 5y ago
  XFS also does checksums, but only for metadata.
- 3np 5y ago
  It depends on the workload. With insufficent memory and/or latency, ZFS can perform orders of magnitude worse than ext4/xfs, even when tuned. (I speak from experience) Not exactly the same, but did you ever try ZFS on a commodity USB stick? It can get really bad when circumstances are not in favor. With not too much concurrency it can perform quite well even under constraints, though!
KaiserPro 5y ago
I've run ZFS on 2/4/8 & 16 gig machines. Yes, more ram makes things a bit faster, as you get a bigger arc cache. however as you point out, 4gigs is more than enough to surface a 1gig connection. Especially on a bunch of SSDs

exhilaration 5y ago

What's the solution for a budget NAS setup with ECC? Stick with Intel/AMD (for now)?

nine_k 5y ago

For that reason, my choice for a home NAS system board was a retired itx server board with a 20W Atom, 8 SATA ports, and ECC memory. Comparable to RPi at compute power, but with vastly better I/O and, again, ECC RAM, because I don't want my data to have a chance to be silently corrupted. I think there can be a niche for a system board like that, only open and based on ARM. IDK if any smaller ARM processors support ECC, though.

bombcar 5y ago
Do you have more details on that - part number or etc? I have an older beast of a machine that I'm thinking of replacing to get power consumption down, but I'd like to keep ECC for ZFS.
- nine_k 5y ago
  I just trawled eBay for "itx c2750". 2550 can also be considered for lower power, or 3750 for higher price and more capacity. Cxx58 parts can be considered instead of Cxx50 parts. Of course all these CPUs are very old, 7-9 years since introduction. But it's ok by me, they are reasonably cold, have 4 or 8 cores, and I'm looking for a cold, silent, and reliable system, not a screaming fast system. At worst, a 10GbE card can be installed into the lone PCIe slot, but I don't expect my rig to saturate even the 1GbE built-in interface. (My board has four of these anyway.) As of motherboard producers, SuperMicro makes the right kind: passive cooling, most / all ports wired, internal USB and / or m.2. Gigabyte made more consumer-oriented variants, but also serviceable.

dpedu 5y ago

ZFS doesn't need ECC ram to work correctly. This is an incorrect but commonly repeated myth. See: https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-y...

ed25519FUUU 5y ago
But wouldn’t it work better with ECC?
segfaultbuserr 5y ago
The original comment did not claim that ZFS specifically needs ECC to run correctly. Quote the OP, > With ZFS allowing for very robust data integrity on disk, the weak point in most budget NAS setups these days is going to be memory errors. You see, it didn't say "no ECC, no ZFS." It merely says: if your goal is to eliminate data corruption, once you've removed filesystem errors from the equation by using ZFS, the next term to eliminate is memory errors - which is entirely correct.
wyager 5y ago
It doesn’t “need” it, but ZFS + ECC is a lot more effective than ZFS + high-BER RAM. Once you fix disk corruption, the next thing you’ll probably worry about is memory corruption.

logifail 5y ago

> the weak point in most budget NAS setups these days is going to be memory errors I use a HPE ProLiant MicroServer Gen8 as a (very) budget home/home-office NAS, I'm really not convinced that memory errors are its weak point(!) It has the default (horrible) Celeron G1610T processor, was upgraded on delivery to 16G of memory, but has 60TB of storage - four internal SATA drives plus additional external USB drives. I've tried and so far failed to get it to run stable 10GbE networking. Just sourced a new (old) ConnectX-3 card from eBay to try again...

gothroach 5y ago
I use an i3-4370 with 32G of ECC in a Supermicro 4u case as my ZFS NAS - I got it new for the purpose and it's still running strong seven years later. The only changes I've made have been doubling the RAM from 16G, adding in new storage as needed, and adding a dual 10gbit card when I upgraded my network. I tried a couple of 10gbit cards, but ended up getting a half dozen used Intel X520-DA2 cards for dirt cheap. I've not had any issue with these, and my NAS can easily saturate at least one link. The cards have been reliable in every machine I've tried and can be found for $75-100 - not too bad for dual port 10gbit! As a side note, I was surprised to find that i3 supports ECC.
forty 5y ago
I use a MicroServer too (an older one). They have ECC memory, so you should indeed be fine on this point. But price and electricity consumption are much higher than a RPi. I agree that the CPU is the weak point (mine has a turion n40l which is slow as hell).
- logifail 5y ago
  > mine has a turion n40l Heh, I have two of those older models lurking in the garage from some older experimental stuff. They were at one point configured as a DRBD two-node cluster, which was fun, although performance was dire.

blibble 5y ago

I've run ZFS on my personal (read: cheap) hardware for several hundred machine years and never had a problem with standard RAM (and issues would be detectable with the monthly scrub) obviously ECC is better but you're no worse off than you would be with mdraid or some awful hardware controller

wyager 5y ago
> and issues would be detectable with the monthly scrub Depends where the corruption happened. Scrub can’t detect data that got corrupted before the checksum was taken.

denimnerd42 5y ago

A ryzen system with ecc can be had for less than $400 build your own.

KaiserPro 5y ago
yes, but even the smallest ryzen sucks down more power and the system is physically bigger. but its way more powerful so can do more stuff.
- denimnerd42 5y ago
  Yeah low power single board computer systems with all the right features tend to be very expensive and difficult to source without large orders. A ryzen system at 20 or 40 watts idle is something you can probably put up with for $400

1MachineElf 5y ago

I completely agree with you. ZFS has led the way in data integrity and ergonomics, and it's time we got it in a small form factor with reliable RAM. Many commenters have stated that it would be better to go with a Ryzen/ECC or some server-grade Atom board. While that may be true from a performance standpoint, it is not true if your priorities are low power usage and saving space. I have several big beefy servers to run ZFS on my network. What I am missing is something tiny and quiet that can do the same job at a small scale just as well while the big machines are turned off. I am not concerned about maintaining fast I/O or running some RAM-hogging zraid6. I just want 4 separate SSDs running 4 separate spools that I can read/write to anytime and later sync with the big machines when they're online. The even the 4GB RPi would perform that fine - I just don't see a point if the RAM is not ECC.

dragontamer 5y ago
> low power usage How about turning the NAS on-and-off as you need it? There's no need to run an "always on" computer unless you want to. If you want to run an IRC bouncer or a webserver, then buy a Rasp. Pi for that. The NAS is primarily storage, which doesn't necessarily need to be always on. (Besides, turning the computer off is the ultimate security move. You can't get that data hacked if the computer isn't even on) ------------ > I have several big beefy servers to run ZFS on my network. What I am missing is something tiny and quiet that can do the same job at a small scale just as well while the big machines are turned off. Ever try just... sticking multiple small-and-cheap hard drives into the clients? Its not too hard to make redundant RAID1-like storage spaces (Windows-clients) or RAID1-like LVM (Linux-clients).

kalleboo 5y ago

> the weak point in most budget NAS setups these days is going to be memory errors In practice, the weak point in most budget NAS setups (1) no backups "because RAID" and then getting hit by a cryptolocker (2) drives dying

dragontamer 5y ago
> (2) drives dying Is that really a problem? I'm still able to find 1TB new hard drives at my local microcenter for just $30. So its not really that expensive to set up a 4x drive RAID10 or RAID6 array. 4x 1TB in RAID10 or RAID6 == 2TB with redundancy for $120ish. Go up to 4TB drives ($100 each) and you'll get 8TB capacity for maybe $400ish. Prices suck right now, even for hard drives. (I used to get 6TB hard drives for $150... but the 4TB point still seems reasonable). > (1) no backups and cryptolockers That seems to be easily solved by regular ZFS-snapshots.
- toast0 5y ago
  Hard drives do die, and while many (possibly most?) deaths are random materials failure, some deaths are correlated to batches and some are firmware problems which are often correlated. Avoiding these is hard. Ideally, you want to get drives from different batches and offset power on times by about a week per drive. Assuming you will notice a failure, obtain a replacement, and replace the failed drive within a week. If that's not realistic, you should wait even longer between drive power on, which makes setup a very long process. It's also hard to ensure drives are from different batches unless you buy in person or get different brands, but there's not four brands to buy from anymore. It's a little easier to do best practices when you replace drives (to get more capacity for example)... Buy one new drive once every other month, and you probably will get different batches, and you'll most likely have sufficient time to replace drives without running into the same firmware bug on all the drives.
- tinus_hn 5y ago
  If only all these mega corporations that get hit by crypto lockers had heard the magic bullet of ZFS! /s

seltzered_ 5y ago

> The PCEngines APU2 has an ECC option, but it does not have the appropriate I/O from looking at https://pcengines.ch/apu2.htm https://pcengines.ch/apu2.htm , perhaps one could bridge from the miniPCI express to a SAN eSATA tower? (rough example: https://hobo.house/2016/02/03/replacing-failed-linux-raid-disks-with-sans-digital-8bay-esata-tower/ https://hobo.house/2016/02/03/replacing-failed-linux-raid-di... )

colechristensen 5y ago

Ok, I see this a lot. Does anyone have real world experience with meaningful corruption which could be pointed at memory errors? A certain sort of person is very interested in ECC and I have a lot of doubt that it actually causes anybody trouble commensurate with the attention it gets (unless you are Google, and you aren't Google) Even if a few bits are flipped in a media file once during transfer or even permanently... or if the OS state of my storage box gets corrupted and I have to reboot it... so what?

wyager 5y ago
RAM BER is shockingly high, like several bits per GB per year. Dozens or hundreds of data corruptions per year does not meet my requirements for even a home NAS, even if most are transient.
- colechristensen 5y ago
  Well what are your requirements? What are the outcomes you think are going to happen which you are trying to avoid? The "error rate" of, say, a dust mote on my TV screen would seem to be orders of magnitude higher than a few flipped bits per GB*year. I have quite a bit of doubt about how many bit errors would ever hit A) during a time in which I am actually retrieving data (or much much less likely saving) and B) actually hits the surface the data is resident in memory and C) has an impact that raises to a perceptual level.