3 ms·

There has not ever been a reason for memory to be correlated with storage capacity nor any reason to believe that such a correlation ought to exist. Nobody eve

There has not ever been a reason for memory to be correlated with storage capacity nor any reason to believe that such a correlation ought to exist. Nobody ever said well I plugged an 20TB external hard drive so I better plug in a few more sticks of RAM so that works. Dedup needs RAM in proportion to storage because for each duplicate block it maintains an entry in an in memory table.

lazide 5y ago
You uh just kind of contradicted yourself? All file systems have metadata which is good to keep in memory. Building several 50+TB NAS boxes recently, it isn’t just ZFS either. And it isn’t some sort of linear performance penalties sometimes when you don’t have enough RAM. It can be kernel panics, exponential decay in performance, etc.
- michaelmrose 5y ago
  I didn't contradict myself at all virtually nobody needs dedup its not remotely worth the RAM cost for 99.9% of users. Can you quantify what you are saying. What OS/filesystem? What minimum RAM requirements for what amount of storage?
  - lazide 5y ago
    That’s a good point - I’ve seen free memory drop every time I’ve built the larger file systems (and not from just the cache), but I never tried to quantify it. And I don’t see any good stats or notes on it. seems like no one is building these larger systems on boxes small enough for it to matter, or at least google isn’t finding it.
    - michaelmrose 5y ago
      Another way of saying this is that RAM usage doesn't meaningfully scale with storage size in scope of storage systems encountered by actual non theoretical people because the minimum ram available on any system one encounters is sufficient to service the amount of storage that it is possible to use on said system.
  - AdrianB1 5y ago
    The main case of deduplication that I know is hosting many virtual machines with the same OS. If that OS is Windows (explains why VM and not containers), there are dozens of GB of data that is duplicated per VM. It is not 'nobody', even if it is not common and not always worth.
magicalhippo 5y ago
> There has not ever been a reason for memory to be correlated with storage capacity nor any reason to believe that such a correlation ought to exist. However specific implementations can indeed have memory requirements that scale in relation to storage capacity. For example, if the implementation keeps the bitmap of free space in memory, then more storage = larger bitmap = more memory required. There's been several attempts in ZFS to reduce memory overhead. I'm pretty sure that if you took a decade old version of ZFS you'd struggle to run it on a system with 512MB RAM.
- michaelmrose 5y ago
  At present 512MB of RAM is notable in how ridiculously tiny it is and 2TB is still an acceptable amount of storage. Without resorting to decades obsolete software can you put a pin on exactly how much storage it would take to render that tiny amount of RAM unusable and then explain how much storage it would take to render a machine with 4GB of RAM likewise unusable so that we may demonstrate memory usage scaling with storage?
  - magicalhippo 5y ago
    My point was merely that your blanket statement doesn't really hold water, since any actual memory requirements by a filesystem would be implementation specific. I will agree that ZFS should handle large pools once you clear the ~fixed minimum memory requirement.
- radiowave 5y ago
  My first ZFS box (running OpenSolaris 2008.11) ran on 512MB, sometimes including a Gnome 2 desktop. It wasn't fast - I didn't need it to be - but it absolutely did work.
  - magicalhippo 5y ago
    Interesting. I started with ZFS around 2009, and I recall several people struggling to us it on a system with less than 2GB, even after messing with tuneables. That was on FreeBSD though, so maybe implementation specific.
porkgymnastics 5y ago

lazide 5y ago

You uh just kind of contradicted yourself? All file systems have metadata which is good to keep in memory. Building several 50+TB NAS boxes recently, it isn’t just ZFS either. And it isn’t some sort of linear performance penalties sometimes when you don’t have enough RAM. It can be kernel panics, exponential decay in performance, etc.

michaelmrose 5y ago
I didn't contradict myself at all virtually nobody needs dedup its not remotely worth the RAM cost for 99.9% of users. Can you quantify what you are saying. What OS/filesystem? What minimum RAM requirements for what amount of storage?
- lazide 5y ago
  That’s a good point - I’ve seen free memory drop every time I’ve built the larger file systems (and not from just the cache), but I never tried to quantify it. And I don’t see any good stats or notes on it. seems like no one is building these larger systems on boxes small enough for it to matter, or at least google isn’t finding it.
  - michaelmrose 5y ago
    Another way of saying this is that RAM usage doesn't meaningfully scale with storage size in scope of storage systems encountered by actual non theoretical people because the minimum ram available on any system one encounters is sufficient to service the amount of storage that it is possible to use on said system.
- AdrianB1 5y ago
  The main case of deduplication that I know is hosting many virtual machines with the same OS. If that OS is Windows (explains why VM and not containers), there are dozens of GB of data that is duplicated per VM. It is not 'nobody', even if it is not common and not always worth.

magicalhippo 5y ago

> There has not ever been a reason for memory to be correlated with storage capacity nor any reason to believe that such a correlation ought to exist. However specific implementations can indeed have memory requirements that scale in relation to storage capacity. For example, if the implementation keeps the bitmap of free space in memory, then more storage = larger bitmap = more memory required. There's been several attempts in ZFS to reduce memory overhead. I'm pretty sure that if you took a decade old version of ZFS you'd struggle to run it on a system with 512MB RAM.

michaelmrose 5y ago
At present 512MB of RAM is notable in how ridiculously tiny it is and 2TB is still an acceptable amount of storage. Without resorting to decades obsolete software can you put a pin on exactly how much storage it would take to render that tiny amount of RAM unusable and then explain how much storage it would take to render a machine with 4GB of RAM likewise unusable so that we may demonstrate memory usage scaling with storage?
- magicalhippo 5y ago
  My point was merely that your blanket statement doesn't really hold water, since any actual memory requirements by a filesystem would be implementation specific. I will agree that ZFS should handle large pools once you clear the ~fixed minimum memory requirement.
radiowave 5y ago
My first ZFS box (running OpenSolaris 2008.11) ran on 512MB, sometimes including a Gnome 2 desktop. It wasn't fast - I didn't need it to be - but it absolutely did work.
- magicalhippo 5y ago
  Interesting. I started with ZFS around 2009, and I recall several people struggling to us it on a system with less than 2GB, even after messing with tuneables. That was on FreeBSD though, so maybe implementation specific.

porkgymnastics 5y ago