3 ms·

> ZFS needs around 1-5GB of ram per TB of storage ...per unique TB written to datasets with deduplication enabled, of course, not the pool as a whole (even tho

> ZFS needs around 1-5GB of ram per TB of storage ...per unique TB written to datasets with deduplication enabled, of course, not the pool as a whole (even though that ram cost is indeed at the pool level). there are plenty of useful things to be done with zfs dedupe at practically no cost if you design for it ahead of time. unfortunately for some reason people parachute into every thread where zfs comes up to say you can never ever turn it on, so people are taught not to even try.

scheme271 1mo ago
Just curious but what would be a good way to design for it ahead of time?
- nagaiaida 1mo ago
  mainly i just mean dataset configuration and arrangement within the filesystem in general as it pertains to whatever you're actually doing. if you can reliably send writes with different storage characteristics (deduplicability, compressibility, etc.) to different datasets configured accordingly, you don't have to deal with the drawbacks of trying to dedupe service logs or installed packages (which will just slowly inflate the dedupe tables for no real benefit) or compress media/encrypted files (admittedly you can usually just bail out early compressing so this side doesn't matter much but imagine the next really good compression algorithm can't fail fast for some reason). so as long as you only enable dedupe where it will actually help you because you designated a dataset for some highly duplicated purpose (and a bunch of transient writes won't harm you by growing the tables needlessly), there's really no reason not to go for it unless tight write latency constraints are also in play. for instance, i have a pool with plenty of datasets that have dedupe off and only a crucial few with it enabled: the ones that are nfs-mounted media storage volumes for fediverse instance containers that would otherwise all duplicate all media they see in common, which is most media posted to any of them and a good bit of the remote media received by any of them. the size of the dedupe tables in memory is purely a function of the number of blocks written to those few datasets and i expect to either sunset this laptop from 2012 or migrate the pool to a fresh one long before i would be worried about the memory cost of deduplication swelling a few orders of magnitude to where i'd have to care.

scheme271 1mo ago

Just curious but what would be a good way to design for it ahead of time?

nagaiaida 1mo ago
mainly i just mean dataset configuration and arrangement within the filesystem in general as it pertains to whatever you're actually doing. if you can reliably send writes with different storage characteristics (deduplicability, compressibility, etc.) to different datasets configured accordingly, you don't have to deal with the drawbacks of trying to dedupe service logs or installed packages (which will just slowly inflate the dedupe tables for no real benefit) or compress media/encrypted files (admittedly you can usually just bail out early compressing so this side doesn't matter much but imagine the next really good compression algorithm can't fail fast for some reason). so as long as you only enable dedupe where it will actually help you because you designated a dataset for some highly duplicated purpose (and a bunch of transient writes won't harm you by growing the tables needlessly), there's really no reason not to go for it unless tight write latency constraints are also in play. for instance, i have a pool with plenty of datasets that have dedupe off and only a crucial few with it enabled: the ones that are nfs-mounted media storage volumes for fediverse instance containers that would otherwise all duplicate all media they see in common, which is most media posted to any of them and a good bit of the remote media received by any of them. the size of the dedupe tables in memory is purely a function of the number of blocks written to those few datasets and i expect to either sunset this laptop from 2012 or migrate the pool to a fresh one long before i would be worried about the memory cost of deduplication swelling a few orders of magnitude to where i'd have to care.