10 ms·

RAM? Everytime I looked into setting up a freenas box, every hardware guide insisted that ungodly amounts of absolutely-has-to-be-ECC RAM was essential, and I

by auxym 5y ago

RAM? Everytime I looked into setting up a freenas box, every hardware guide insisted that ungodly amounts of absolutely-has-to-be-ECC RAM was essential, and I just gave up at that point.

n8ta 5y ago
The freenas hardware requirements themselves say "8 GB RAM (ECC recommended but not required)" https://www.freenas.org/hardware-requirements/ https://www.freenas.org/hardware-requirements/ I myself use freenas with 16GB of non-ECC ram. Of course it is possible to have a bit flip in memory that is then dutifully stored incorrectly by ZFS to disk, but this was a possibility without ZFS as well. I've actually been waiting for this feature for since I first setup my pool. It seemed theoretically possible we were just waiting for an implementation.
dsr_ 5y ago
Neither quantity nor ECC is essential. ZFS defaults to assuming it is the primary reason for your box to exist, but it only takes two lines to define more reasonable RAM usage: zfs_arc_min and zfs_arc_max. On a NAS type server, I would think setting the max to half of your RAM is reasonable. Maybe 3/4 if you never do anything except storage. ECC is not recommended because ZFS has some kind of special vulnerability without it; ECC is recommended because ZFS has taken care of all the more likely chances of undetectable corruption, so that's the next step.
- fpoling 5y ago
  It is not that simple regarding ECC. Since ZFS uses more memory, the probability of hitting a memory bug is simply higher with it.
  - amarshall 5y ago
    But it doesn’t really use more memory. The ARC gives the impression of high memory usage because it’s different than the OS page cache and usually called out explicitly and not ignored in many monitoring tools like the OS cache is. Linux—without ZFS—will happily consume nearly all RAM with any filesystem if enough data is read and written.
    - dsr_ 5y ago
      This is correct. Any filesystem using the kernel's filesystem cache will do this, too. For a long running, non-idle system, a good rule of thumb is that all RAM not being actively used is being used by evictable caching.
      - zerd 5y ago
        A colleague who was used to other UNIXes was transitioning to Linux for a database. He saw in free that used was more at more than 90%, so he added more ram. But to his surprise it was still using 90%! He kept adding ram. I told him that he had to subtract the buffer and cached values (this was before free had the Available column).
        Brian-Puccio 5y ago
        https://www.linuxatemyram.com/ https://www.linuxatemyram.com/ is one of my favorite single-serving sites.
        mark-wagner 5y ago
        Before the Available column there was the -/+ buffers/cache line that provided the same information. Maybe it was too confusing. total used free shared buffers cached Mem: 12286456 11715372 571084 0 81912 6545228 -/+ buffers/cache: 5088232 7198224 Swap: 24571408 54528 24516880
IgorPartola 5y ago
Heh so you have that backwards. All RAM should be ECC if you care about what’s stored in it. It’s not a ZFS requirement, it’s just that ZFS specifically cares about data integrity so it advises you to use ECC RAM. But it’s not like any other file system is immune from random RAM corruption: it’s not, it just won’t tell you about it.
handrous 5y ago
The "you need at least 32GB of memory and it has to be ECC, or don't even bother trying to use ZFS" crowd has done some serious harm to ZFS adoption. Sure, that's what you need if you want excellent data integrity guarantees and to use all of ZFS' advanced features. If you're fine with merely way-better-than-most-other-filesystems data integrity guarantees and using only most of ZFS' advanced features, you don't need those.
- tombert 5y ago
  I really don't know where the "You gotta have ECC RAM!" thing started. I've been running a ZFS RAID on Nvidia Jetson Nanos for years now and haven't had any issues at all with data integrity. I don't see why ZFS would be more prone to data integrity issues spawning from a lack of ECC than any other filesystem.
  - radiowave 5y ago
    Relevant quote from one of ZFS's primary designers, Matt Ahrens: “There's nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem. ... I would simply say: if you love your data, use ECC RAM. Additionally, use a filesystem that checksums your data, such as ZFS."
    - tombert 5y ago
      Yeah, I remember reading that a few years ago. If I were running a server farm or something, then yeah, I'd probably use ECC memory, but I think if you're running a home server, then the argument that ZFS necessitates ECC more than Ext4 or Btrfs or XFS or whatever doesn't really seem to be accurate.
      - oarsinsync 5y ago
        > the argument that ZFS necessitates ECC more than Ext4 or Btrfs or XFS or whatever doesn't really seem to be accurate Agreed. > If I were running a server farm or something, then yeah, I'd probably use ECC memory, but I think if you're running a home server Then you should still use ECC RAM, regardless of what filesystem you're using. No, really. ECC matters (https://news.ycombinator.com/item?id=25622322 https://news.ycombinator.com/item?id=25622322) generally.
simcop2387 5y ago
You really only end up needing that if and only if you're also going to do live deduplication of large amounts of data. Very few people actually need that, just using compression with lz4 or zstd depending on your needs will suffice for just about everyone and perform better. the ECC argument is probably about a 50/50 kind of thing, you can get away without it and ZFS will do it's best to detect and prevent issues but if the data was flipped before it was given to ZFS then there's nothing anyone can do. You might get some false positives when reading data back if you got some flaky ram but as long as you have parity or redundancy on the disks then things should still get read correctly even if a false problem is detected. That might mean you want to run a scrub (essentially ZFS's version of fsck) more often to look for potential issues but it shouldn't fundamentally be a big deal. If you end up wanting 24/7 highly available storage that won't blip out occasionally you'll probably really want the ECC ram but if you're fine with having to reboot it occasionally or tell it to repair problems that it thinks were there (but weren't because the disk is fine but the ram wasn't) then you should be fine. The extra checksums and data that ZFS can use for all this can make it really robust even on bad hardware. I had a bios update cause some massive PCIE bus issues that I didn't realize were going on for a bit and ZFS kept all my data in good condition even though writes were sometimes just never happening because of ASPM causing issues with my controller card.
colechristensen 5y ago
ZFS likes RAM and uses it to get better performance (and don't think about using dedup without huge ram), but you don't need it and can change the defaults. ECC tends to attract zealots after a perfect error-free existence which ECC does tend towards but doesn't deliver, it just reduces errors. I personally don't care about a tiny amount of bit rot (zfs will prevent most of this) and rebooting my storage machine now and then. You can run ZFS/freenas on a crappy old machine and you'll be just fine as long as you aren't hosting storage for dozens of people and you aren't a digital archivist trying to keep everything for centuries. Real advice: * Mirrored vdevs perform way better than raidz, I don't think the storage gain is worth it until you have dozens of drives * Dedup isn't worth it * Enable lz4 compression everywhere * Have a hot spare * You can increase performance by adding a vdev set and by adding RAM * Use drives with the same capacity
- InvaderFizz 5y ago
  > Dedup isn't worth it To add to that, ZFS dedup is a lie and you should forget its existence unless you have a very specific scenario of being a SAN with a massive amount of RAM, and even then, you had better be damn sure. I really wish ZFS had either an option to store the Dedup Table on a NVMe like Optane, or to do an offline deduplication job.
  - rincebrain 5y ago
    It does have the former, these days - the "allocation_classes" feature lets you make the permanent home of certain subsets of data on "special" vdevs - which includes methods of specifying "store dedup table there". Now, that becomes the only place entries on it are stored, so you best make it redundant if you don't want to lose your pool from a single NVMe failing, but the feature is there. The latter I would predict seeing approximately when the sun burns out, on ZFS. It _really_ doesn't like the idea of data changing locations retroactively.
    - InvaderFizz 5y ago
      Thanks for this. I completely missed this feature in the run up to 0.8. I'm going to have to do some test setups with this.
UI_at_80x24 5y ago
As always, it depends on your use-case. I have several file-servers all use ZFS exclusively. and 10x that number of servers using ZFS as the system FS. Rule of thumb that I like: 1GB RAM/TB of storage. This seems to give me the best bang-for-our-buck. For a small (under 20) number of office users, doing general 'office' stuff, using Samba, it's overkill. For large media shares with heavy editor access, and heavy strains on the network, it's a minimum. Depends on what the server is serving. DeDUP is a different story. The RAM is used to store the frequently accessed data. If you are using DeDUP you fill the motherboard with as much RAM as will fit. NO EXCEPTIONS! This may have been the line of thinking that scared you away from it. I have a 100TB server that is just used for writing data to and is never read from (sequential file back-ups before it's moved to "long term storage"). It has 8GB of RAM, and is barely touched. I also have a 20TB server with 2TB of RAM, that keeps the RAM maxed out with DeDUP usage. ECC: It's insurance, and it's worth it.
- rincebrain 5y ago
  That's not precisely why dedup needs gobs of RAM. (If you already know this distinction, I apologize, I just want to make sure people reading this do.) You effectively (unless you use allocation classes) need to keep the entire DDT in RAM all the time if you don't want any write to a dedup-enabled dataset to potentially require blocking on reading the relevant segment from spinning disks into RAM (thus tanking performance even worse than dedup normally does). It's not really related to the mechanisms in the rest of ZFS for keeping {frequently,recently} used data cached in RAM.
ahofmann 5y ago
Others have said good things (ECC is good by itself, has not much to do with ZFS) and it is actually quite easy to check if you need much RAM for ZFS. Start a (Linux) VM with a few hundred megabytes of RAM and run ZFS an on it. Of course, it will not be as performant as having a lot of RAM. But it will not crash, or hang or be unusable in one way or another. Sources: - https://www.reddit.com/r/DataHoarder/comments/3s7vrd/so_you_think_zfs_needs_a_ton_of_ram_for_a_simple/ https://www.reddit.com/r/DataHoarder/comments/3s7vrd/so_you_... - https://www.reddit.com/r/homelab/comments/8s6r2r/what_exactly_is_sacrificed_with_low_ram_when/e0x71ba/ https://www.reddit.com/r/homelab/comments/8s6r2r/what_exactl... - My own tests with around 8 TB ZFS data in a Linux vm with 256 MB RAM.
seized 5y ago
FreeNAS is excellent in many ways. Except that weird gospel their forum people have. ZFS only needs a lot of RAM if deduplication is enabled. And it shouldn't be for most use cases, or only enabled on one dataset that benefits from it. Many ZFS installs are fine on 8GB or less. ECC RAM is better but not required. The idea is to catch memory errors, hence ECC is better.

by auxym 5y ago

RAM? Everytime I looked into setting up a freenas box, every hardware guide insisted that ungodly amounts of absolutely-has-to-be-ECC RAM was essential, and I just gave up at that point.

n8ta 5y ago

The freenas hardware requirements themselves say "8 GB RAM (ECC recommended but not required)" https://www.freenas.org/hardware-requirements/ https://www.freenas.org/hardware-requirements/ I myself use freenas with 16GB of non-ECC ram. Of course it is possible to have a bit flip in memory that is then dutifully stored incorrectly by ZFS to disk, but this was a possibility without ZFS as well. I've actually been waiting for this feature for since I first setup my pool. It seemed theoretically possible we were just waiting for an implementation.

dsr_ 5y ago

Neither quantity nor ECC is essential. ZFS defaults to assuming it is the primary reason for your box to exist, but it only takes two lines to define more reasonable RAM usage: zfs_arc_min and zfs_arc_max. On a NAS type server, I would think setting the max to half of your RAM is reasonable. Maybe 3/4 if you never do anything except storage. ECC is not recommended because ZFS has some kind of special vulnerability without it; ECC is recommended because ZFS has taken care of all the more likely chances of undetectable corruption, so that's the next step.

fpoling 5y ago
It is not that simple regarding ECC. Since ZFS uses more memory, the probability of hitting a memory bug is simply higher with it.
- amarshall 5y ago
  But it doesn’t really use more memory. The ARC gives the impression of high memory usage because it’s different than the OS page cache and usually called out explicitly and not ignored in many monitoring tools like the OS cache is. Linux—without ZFS—will happily consume nearly all RAM with any filesystem if enough data is read and written.
  - dsr_ 5y ago
    This is correct. Any filesystem using the kernel's filesystem cache will do this, too. For a long running, non-idle system, a good rule of thumb is that all RAM not being actively used is being used by evictable caching.
    - zerd 5y ago
      A colleague who was used to other UNIXes was transitioning to Linux for a database. He saw in free that used was more at more than 90%, so he added more ram. But to his surprise it was still using 90%! He kept adding ram. I told him that he had to subtract the buffer and cached values (this was before free had the Available column).
      - Brian-Puccio 5y ago
        https://www.linuxatemyram.com/ https://www.linuxatemyram.com/ is one of my favorite single-serving sites.
      - mark-wagner 5y ago
        Before the Available column there was the -/+ buffers/cache line that provided the same information. Maybe it was too confusing. total used free shared buffers cached Mem: 12286456 11715372 571084 0 81912 6545228 -/+ buffers/cache: 5088232 7198224 Swap: 24571408 54528 24516880

IgorPartola 5y ago

Heh so you have that backwards. All RAM should be ECC if you care about what’s stored in it. It’s not a ZFS requirement, it’s just that ZFS specifically cares about data integrity so it advises you to use ECC RAM. But it’s not like any other file system is immune from random RAM corruption: it’s not, it just won’t tell you about it.

handrous 5y ago

The "you need at least 32GB of memory and it has to be ECC, or don't even bother trying to use ZFS" crowd has done some serious harm to ZFS adoption. Sure, that's what you need if you want excellent data integrity guarantees and to use all of ZFS' advanced features. If you're fine with merely way-better-than-most-other-filesystems data integrity guarantees and using only most of ZFS' advanced features, you don't need those.

tombert 5y ago
I really don't know where the "You gotta have ECC RAM!" thing started. I've been running a ZFS RAID on Nvidia Jetson Nanos for years now and haven't had any issues at all with data integrity. I don't see why ZFS would be more prone to data integrity issues spawning from a lack of ECC than any other filesystem.
- radiowave 5y ago
  Relevant quote from one of ZFS's primary designers, Matt Ahrens: “There's nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem. ... I would simply say: if you love your data, use ECC RAM. Additionally, use a filesystem that checksums your data, such as ZFS."
  - tombert 5y ago
    Yeah, I remember reading that a few years ago. If I were running a server farm or something, then yeah, I'd probably use ECC memory, but I think if you're running a home server, then the argument that ZFS necessitates ECC more than Ext4 or Btrfs or XFS or whatever doesn't really seem to be accurate.
    - oarsinsync 5y ago
      > the argument that ZFS necessitates ECC more than Ext4 or Btrfs or XFS or whatever doesn't really seem to be accurate Agreed. > If I were running a server farm or something, then yeah, I'd probably use ECC memory, but I think if you're running a home server Then you should still use ECC RAM, regardless of what filesystem you're using. No, really. ECC matters (https://news.ycombinator.com/item?id=25622322 https://news.ycombinator.com/item?id=25622322) generally.

simcop2387 5y ago

You really only end up needing that if and only if you're also going to do live deduplication of large amounts of data. Very few people actually need that, just using compression with lz4 or zstd depending on your needs will suffice for just about everyone and perform better. the ECC argument is probably about a 50/50 kind of thing, you can get away without it and ZFS will do it's best to detect and prevent issues but if the data was flipped before it was given to ZFS then there's nothing anyone can do. You might get some false positives when reading data back if you got some flaky ram but as long as you have parity or redundancy on the disks then things should still get read correctly even if a false problem is detected. That might mean you want to run a scrub (essentially ZFS's version of fsck) more often to look for potential issues but it shouldn't fundamentally be a big deal. If you end up wanting 24/7 highly available storage that won't blip out occasionally you'll probably really want the ECC ram but if you're fine with having to reboot it occasionally or tell it to repair problems that it thinks were there (but weren't because the disk is fine but the ram wasn't) then you should be fine. The extra checksums and data that ZFS can use for all this can make it really robust even on bad hardware. I had a bios update cause some massive PCIE bus issues that I didn't realize were going on for a bit and ZFS kept all my data in good condition even though writes were sometimes just never happening because of ASPM causing issues with my controller card.

colechristensen 5y ago

ZFS likes RAM and uses it to get better performance (and don't think about using dedup without huge ram), but you don't need it and can change the defaults. ECC tends to attract zealots after a perfect error-free existence which ECC does tend towards but doesn't deliver, it just reduces errors. I personally don't care about a tiny amount of bit rot (zfs will prevent most of this) and rebooting my storage machine now and then. You can run ZFS/freenas on a crappy old machine and you'll be just fine as long as you aren't hosting storage for dozens of people and you aren't a digital archivist trying to keep everything for centuries. Real advice: * Mirrored vdevs perform way better than raidz, I don't think the storage gain is worth it until you have dozens of drives * Dedup isn't worth it * Enable lz4 compression everywhere * Have a hot spare * You can increase performance by adding a vdev set and by adding RAM * Use drives with the same capacity

InvaderFizz 5y ago
> Dedup isn't worth it To add to that, ZFS dedup is a lie and you should forget its existence unless you have a very specific scenario of being a SAN with a massive amount of RAM, and even then, you had better be damn sure. I really wish ZFS had either an option to store the Dedup Table on a NVMe like Optane, or to do an offline deduplication job.
- rincebrain 5y ago
  It does have the former, these days - the "allocation_classes" feature lets you make the permanent home of certain subsets of data on "special" vdevs - which includes methods of specifying "store dedup table there". Now, that becomes the only place entries on it are stored, so you best make it redundant if you don't want to lose your pool from a single NVMe failing, but the feature is there. The latter I would predict seeing approximately when the sun burns out, on ZFS. It _really_ doesn't like the idea of data changing locations retroactively.
  - InvaderFizz 5y ago
    Thanks for this. I completely missed this feature in the run up to 0.8. I'm going to have to do some test setups with this.

UI_at_80x24 5y ago

As always, it depends on your use-case. I have several file-servers all use ZFS exclusively. and 10x that number of servers using ZFS as the system FS. Rule of thumb that I like: 1GB RAM/TB of storage. This seems to give me the best bang-for-our-buck. For a small (under 20) number of office users, doing general 'office' stuff, using Samba, it's overkill. For large media shares with heavy editor access, and heavy strains on the network, it's a minimum. Depends on what the server is serving. DeDUP is a different story. The RAM is used to store the frequently accessed data. If you are using DeDUP you fill the motherboard with as much RAM as will fit. NO EXCEPTIONS! This may have been the line of thinking that scared you away from it. I have a 100TB server that is just used for writing data to and is never read from (sequential file back-ups before it's moved to "long term storage"). It has 8GB of RAM, and is barely touched. I also have a 20TB server with 2TB of RAM, that keeps the RAM maxed out with DeDUP usage. ECC: It's insurance, and it's worth it.

rincebrain 5y ago
That's not precisely why dedup needs gobs of RAM. (If you already know this distinction, I apologize, I just want to make sure people reading this do.) You effectively (unless you use allocation classes) need to keep the entire DDT in RAM all the time if you don't want any write to a dedup-enabled dataset to potentially require blocking on reading the relevant segment from spinning disks into RAM (thus tanking performance even worse than dedup normally does). It's not really related to the mechanisms in the rest of ZFS for keeping {frequently,recently} used data cached in RAM.

ahofmann 5y ago

Others have said good things (ECC is good by itself, has not much to do with ZFS) and it is actually quite easy to check if you need much RAM for ZFS. Start a (Linux) VM with a few hundred megabytes of RAM and run ZFS an on it. Of course, it will not be as performant as having a lot of RAM. But it will not crash, or hang or be unusable in one way or another. Sources: - https://www.reddit.com/r/DataHoarder/comments/3s7vrd/so_you_think_zfs_needs_a_ton_of_ram_for_a_simple/ https://www.reddit.com/r/DataHoarder/comments/3s7vrd/so_you_... - https://www.reddit.com/r/homelab/comments/8s6r2r/what_exactly_is_sacrificed_with_low_ram_when/e0x71ba/ https://www.reddit.com/r/homelab/comments/8s6r2r/what_exactl... - My own tests with around 8 TB ZFS data in a Linux vm with 256 MB RAM.

seized 5y ago

FreeNAS is excellent in many ways. Except that weird gospel their forum people have. ZFS only needs a lot of RAM if deduplication is enabled. And it shouldn't be for most use cases, or only enabled on one dataset that benefits from it. Many ZFS installs are fine on 8GB or less. ECC RAM is better but not required. The idea is to catch memory errors, hence ECC is better.