4 ms·

> Many remain skeptical of deduplication, which hogs expensive RAM in the best-case scenario. And I do mean expensive: Pretty much every ZFS FAQ flatly declares

by carlob 9y ago

> Many remain skeptical of deduplication, which hogs expensive RAM in the best-case scenario. And I do mean expensive: Pretty much every ZFS FAQ flatly declares that ECC RAM is a must-have and 8 GB is the bare minimum. In my own experience with FreeNAS, 32 GB is a nice amount for an active small ZFS server, and this costs $200-$300 even at today’s prices. I use nas4free with much less ram…

eriknstr 9y ago
The massive amounts of RAM recommendation is if you need to do deduplication. Are you doing that? If not then you don't need a lot of RAM. Between the low cost of storage, and alternative solutions for deduplicating data I personally don't use the built-in deduplication functionality of ZFS for my zpools. Might come down to what sorts of data you are storing, though.
- sfoskett 9y ago
  Yup. This. Now that I have 32 GB of RAM in my FreeNAS box I decided I really didn't need dedupe after all. I just don't have that much duplicated data, and I've got 60 TB of HDDs in the box. So I use the RAM for VMs instead!
  - agumonkey 9y ago
    Yeah dedup is mostly useful for massively multiusers setups, say mail, file sharing, etc etc. For small setups you can do a `fdupes` pass once a day and fix it yourself I suppose.
293984j29384 9y ago
I remember that time I turned on dedup on our 45drive box with 32GB of ram and 140TB~ of data..
- mioelnir 9y ago
  ...and nothing happened because it does not retroactively dedup existing data? Over time it would become horrendously slow, I agree, since you have too little RAM by a factor of at least 8.
  - snuxoll 9y ago
    And this is why ixSystems disables access to the dedup switch on their TrueNAS systems without contacting support to enable it. I've got 30TB of small files stored on our TrueNAS system at work, there's no benefit to enabling it but if someone decided to toggle it we'd quickly learn there isn't enough RAM in the world to handle billions of 1-16KB files....

by carlob 9y ago

eriknstr 9y ago

The massive amounts of RAM recommendation is if you need to do deduplication. Are you doing that? If not then you don't need a lot of RAM. Between the low cost of storage, and alternative solutions for deduplicating data I personally don't use the built-in deduplication functionality of ZFS for my zpools. Might come down to what sorts of data you are storing, though.

sfoskett 9y ago
Yup. This. Now that I have 32 GB of RAM in my FreeNAS box I decided I really didn't need dedupe after all. I just don't have that much duplicated data, and I've got 60 TB of HDDs in the box. So I use the RAM for VMs instead!
- agumonkey 9y ago
  Yeah dedup is mostly useful for massively multiusers setups, say mail, file sharing, etc etc. For small setups you can do a `fdupes` pass once a day and fix it yourself I suppose.

293984j29384 9y ago

I remember that time I turned on dedup on our 45drive box with 32GB of ram and 140TB~ of data..

mioelnir 9y ago
...and nothing happened because it does not retroactively dedup existing data? Over time it would become horrendously slow, I agree, since you have too little RAM by a factor of at least 8.
- snuxoll 9y ago
  And this is why ixSystems disables access to the dedup switch on their TrueNAS systems without contacting support to enable it. I've got 30TB of small files stored on our TrueNAS system at work, there's no benefit to enabling it but if someone decided to toggle it we'd quickly learn there isn't enough RAM in the world to handle billions of 1-16KB files....