4 ms·

ZFS has de-duplication features however the amount of RAM required for the lookup tables can become massive and I would generally not recommend using it for lar

by Rapzid 12y ago

ZFS has de-duplication features however the amount of RAM required for the lookup tables can become massive and I would generally not recommend using it for large-ish backup systems. Does this get around the massive lookup tables somehow? Edit: The README addresses this under scalability. Apparently the overhead is much smaller than for ZFS.

Dylan16807 12y ago
And even if you give ZFS enough memory, its access pattern becomes so randomized that it's impossible to get decent throughput on hard drives. Hopefully this system does not suffer from that.
- comboy 12y ago
  > And even if you give ZFS enough memory, its access pattern becomes so randomized that it's impossible to get decent throughput on hard drives. Any idea if the slowdown also happens for SSDs (with enough memory)?
  - Dylan16807 12y ago
    I've never really tried it on SSD, so I can't say. I just know that I couldn't even get small data sets (sub 100 GB) to perform reasonably with dedup. I do know that L2ARC on SSD didn't help, and ZIL never got used.
benjohnson 12y ago
For ZFS, putting in a 128GB SSD for the L2ARC solved the problem for me even though I only have 2GB of Ram for a 12TB ZFS pool.