3 ms·
It's cheaper than the memory upgrade you need to support dedup properly.
by jandrese 7y ago
It's cheaper than the memory upgrade you need to support dedup properly.
- leiroigh 7y agoNot really. ZFS dedup is just brain-dead stupid. Rabin-Karp chunking plus spatial locality / opportunistic dedup allow you to gracefully scale down: you keep a complete table of successfully deduped blocks, plus a small fraction of other blocks, like e.g. smallest 1% of hashes, first blocks in files and final blocks in files. If you have a 12 MB (100 blocks) stretch that matches, you have 63% (1-1/e) chance of discovering one of these blocks as a match, and then use spatial locality to dedup adjacent blocks (duplicate blocks tend to come in bunches).
- jandrese 7y agoThen why does turning on dedup on ZFS suddenly make it chomp through 4GB of memory? Is there some tuneable you're supposed to be setting to make it less insane?