5 ms·

A while back I was considering Freenas and in the documentation/forums I read dire warnings about running ZFS with less than 8 gigabyte of ram. Apparently, this

A while back I was considering Freenas and in the documentation/forums I read dire warnings about running ZFS with less than 8 gigabyte of ram. Apparently, this could actually put your data at risk. I was a bit nonplussed by this. The least I would expect from a file system is that it degrades gracefully. I wonder if anyone knows more about this?

oakwhiz 11y ago
I've been running ZFS on a machine with 768 MB of RAM and there have been no issues. If you set the size of the ARC cache according to your needs and you do not use deduplication then you won't encounter any issues. Deduplication requires a giant hash table and is rarely beneficial.
mkup 11y ago
I run FreeBSD with ZFS on the home server, which primary purpose is storing and processing video files from two surveillance cameras. ZFS pool is created as RAID-1 array of two 2TB HDDs. Server has 2GB DDR2 RAM, but ZFS in-memory cache is restricted to 512MB. OS architecture is x64. ZFS deduplication is turned off. CPU is Intel Atom D510. Not very fast, but it was chosen for low power consumption considering special case when grid power goes off and whole system (including server, cameras and PoE switch) runs from backup DC batteries. I've never had any problems with this setup during 3+ years. It's rock solid and even has extra CPU time to run motion detection software and Transmission (BitTorrent client).
anexprogrammer 11y ago
FreeNAS do seem to recommend ludicrous minimum amounts of RAM, even without dedupe. If you ferret around their forums there seems to have been some history of data loss on low RAM machines. I don't know if it's something they've introduced but I've not seen mention of similar issues around the FreeBSD forums. If you compare with the FreeBSD guide to tuning ZFS, they recommend 1GB and up and give settings for tuning for 768 MB. They mention that in practice dedupe would like upto 5GB / TB, so I've never bothered. 99 out of 100 don't need dedupe anyway - compression will be better. I've never had an issue running FreeBSD + ZFS on a 1GB machine, which was my file server for several years. I ran FreeNAS with 1GB for around a year without issue. I'm now running a HP microserver with 16GB which gives 12GB to ARC.
the_ancient 11y ago
I would have to see proof that is was low ram that caused the data loss. The Recommendations for higher amounts memory comes from the way ZFS Caching works. There are 2 caches, ARC and L2-ARC. L2-ARC has to be defined on a SSD and is optional. ARC will always exist and ZFS will use every bit of free memory you have (unless you change the configuration to limit it) to build the cache.. why ram is faster that is why. So the only scenario I could conceive of that would result in data loss is if you are writing files to the ARC faster than ZFS can commit them to disk but that seems unlikely and ZFS should not allow this anyway. for Enterprise work loads the Recommend amount of Memory is 1GB per TB of storage. This is for best performance, however my home server is 35TB and ran for about 2 years with 8GB, and recently upgraded the system to a new proc, mainboard and 16gb a memory, have had no issues with either. I have about 2-3 Movies streaming from it with about 3-4 devices/computers connected for network file storage. The more RAM you give ZFS the better performance you get (to a point) but I believe this issue is massively over talked about and hyped.
- leonroy 11y ago
  > The more RAM you give ZFS the better performance you get (to a point) but I believe this issue is massively over talked about and hyped. Definitely not hyped as you said in the enterprise space - think some home users see that warning of 1GB per 1TB though and get put off. In my experience we were running multiple vSphere NFS volumes on several large 16x bay FreeNAS systems. We started with 16x 1TB disks and 16GB of RAM. It wasn't until we installed 32GB of RAM before performance became acceptable. Ideally we should really have put in 64GB of RAM but alas Intel's damn Ivy Bridge E3 range maxed out at 32GB. I should stress we were only running 20 virtual machines in total so nothing massively onerous, but that extra RAM made all the difference.
  - the_ancient 11y ago
    >We started with 16x 1TB disks and 16GB of RAM. Type of Drives, did you have a L2 ARC and ZIL... There are alot of reasons why you may have needed the additional ram. If you were using off the Shelf 7200 (or worse 5400) NAS drives then I am not shocked, these drives are for storage not for VM's. need atleast 10K drives for VM storage >Definitely not hyped as you said in the enterprise space - think some home users see that warning of 1GB per 1TB though and get put off. It still kinda is, but 90% of the questions I see posted in various places are about Home and Test Labs, not Enterprise Deployments. you do not need 32GB of RAM for your home file server that is likely serving less than 10 clients.
takeda 11y ago
FreeNAS people seem to mix up what's recommended vs what's required. The whole myth that ZFS requires ECC memory comes from them, but in reality ECC is as much helpful with ZFS as with any other FS. It's not that ZFS will start destroying your data if you don't have ECC. It's just that ZFS provides guarantees, but it can't help much if data is corrupted in memory, but any other filesystem will also be affected in that scenario. Similarly with 8GB. You can actually run ZFS on 1GB, but the more RAM you have the better. ZFS has ARC (Adaptive Replacement Cache[1]) which speeds up read access by storing in RAM most frequently accessed data. Not running ARC won't affect safety of your data, it's just that you'll get worse performance. There are some warts ZFS (and most likely COW file systems) have. For example until ZFS will have a defragmenting tool (doesn't looks like anyone is working on it) you don't want ever to use more than 80% of your disk. If you do, you'll permanently reduce performance of the pool and currently the only fix would be to backup all the data, wipe disk and restore it (this effectively will defragment it). Another issue is that deduplication in ZFS is essentially useless. It requires obscene amount of RAM to be used, and if you enable it with less, it'll drastically reduce performance as well. Instead of deduplication it's better to just enable compression. [1] https://en.wikipedia.org/wiki/Adaptive_replacement_cache https://en.wikipedia.org/wiki/Adaptive_replacement_cache

oakwhiz 11y ago

I've been running ZFS on a machine with 768 MB of RAM and there have been no issues. If you set the size of the ARC cache according to your needs and you do not use deduplication then you won't encounter any issues. Deduplication requires a giant hash table and is rarely beneficial.

mkup 11y ago

I run FreeBSD with ZFS on the home server, which primary purpose is storing and processing video files from two surveillance cameras. ZFS pool is created as RAID-1 array of two 2TB HDDs. Server has 2GB DDR2 RAM, but ZFS in-memory cache is restricted to 512MB. OS architecture is x64. ZFS deduplication is turned off. CPU is Intel Atom D510. Not very fast, but it was chosen for low power consumption considering special case when grid power goes off and whole system (including server, cameras and PoE switch) runs from backup DC batteries. I've never had any problems with this setup during 3+ years. It's rock solid and even has extra CPU time to run motion detection software and Transmission (BitTorrent client).

anexprogrammer 11y ago

FreeNAS do seem to recommend ludicrous minimum amounts of RAM, even without dedupe. If you ferret around their forums there seems to have been some history of data loss on low RAM machines. I don't know if it's something they've introduced but I've not seen mention of similar issues around the FreeBSD forums. If you compare with the FreeBSD guide to tuning ZFS, they recommend 1GB and up and give settings for tuning for 768 MB. They mention that in practice dedupe would like upto 5GB / TB, so I've never bothered. 99 out of 100 don't need dedupe anyway - compression will be better. I've never had an issue running FreeBSD + ZFS on a 1GB machine, which was my file server for several years. I ran FreeNAS with 1GB for around a year without issue. I'm now running a HP microserver with 16GB which gives 12GB to ARC.

the_ancient 11y ago

I would have to see proof that is was low ram that caused the data loss. The Recommendations for higher amounts memory comes from the way ZFS Caching works. There are 2 caches, ARC and L2-ARC. L2-ARC has to be defined on a SSD and is optional. ARC will always exist and ZFS will use every bit of free memory you have (unless you change the configuration to limit it) to build the cache.. why ram is faster that is why. So the only scenario I could conceive of that would result in data loss is if you are writing files to the ARC faster than ZFS can commit them to disk but that seems unlikely and ZFS should not allow this anyway. for Enterprise work loads the Recommend amount of Memory is 1GB per TB of storage. This is for best performance, however my home server is 35TB and ran for about 2 years with 8GB, and recently upgraded the system to a new proc, mainboard and 16gb a memory, have had no issues with either. I have about 2-3 Movies streaming from it with about 3-4 devices/computers connected for network file storage. The more RAM you give ZFS the better performance you get (to a point) but I believe this issue is massively over talked about and hyped.

leonroy 11y ago
> The more RAM you give ZFS the better performance you get (to a point) but I believe this issue is massively over talked about and hyped. Definitely not hyped as you said in the enterprise space - think some home users see that warning of 1GB per 1TB though and get put off. In my experience we were running multiple vSphere NFS volumes on several large 16x bay FreeNAS systems. We started with 16x 1TB disks and 16GB of RAM. It wasn't until we installed 32GB of RAM before performance became acceptable. Ideally we should really have put in 64GB of RAM but alas Intel's damn Ivy Bridge E3 range maxed out at 32GB. I should stress we were only running 20 virtual machines in total so nothing massively onerous, but that extra RAM made all the difference.
- the_ancient 11y ago
  >We started with 16x 1TB disks and 16GB of RAM. Type of Drives, did you have a L2 ARC and ZIL... There are alot of reasons why you may have needed the additional ram. If you were using off the Shelf 7200 (or worse 5400) NAS drives then I am not shocked, these drives are for storage not for VM's. need atleast 10K drives for VM storage >Definitely not hyped as you said in the enterprise space - think some home users see that warning of 1GB per 1TB though and get put off. It still kinda is, but 90% of the questions I see posted in various places are about Home and Test Labs, not Enterprise Deployments. you do not need 32GB of RAM for your home file server that is likely serving less than 10 clients.

takeda 11y ago

FreeNAS people seem to mix up what's recommended vs what's required. The whole myth that ZFS requires ECC memory comes from them, but in reality ECC is as much helpful with ZFS as with any other FS. It's not that ZFS will start destroying your data if you don't have ECC. It's just that ZFS provides guarantees, but it can't help much if data is corrupted in memory, but any other filesystem will also be affected in that scenario. Similarly with 8GB. You can actually run ZFS on 1GB, but the more RAM you have the better. ZFS has ARC (Adaptive Replacement Cache[1]) which speeds up read access by storing in RAM most frequently accessed data. Not running ARC won't affect safety of your data, it's just that you'll get worse performance. There are some warts ZFS (and most likely COW file systems) have. For example until ZFS will have a defragmenting tool (doesn't looks like anyone is working on it) you don't want ever to use more than 80% of your disk. If you do, you'll permanently reduce performance of the pool and currently the only fix would be to backup all the data, wipe disk and restore it (this effectively will defragment it). Another issue is that deduplication in ZFS is essentially useless. It requires obscene amount of RAM to be used, and if you enable it with less, it'll drastically reduce performance as well. Instead of deduplication it's better to just enable compression. [1] https://en.wikipedia.org/wiki/Adaptive_replacement_cache https://en.wikipedia.org/wiki/Adaptive_replacement_cache