4 ms·
Not really. ZFS dedup is just brain-dead stupid. Rabin-Karp chunking plus spatial locality / opportunistic dedup allow you to gracefully scale down: you keep a
by leiroigh 7y ago
Not really. ZFS dedup is just brain-dead stupid. Rabin-Karp chunking plus spatial locality / opportunistic dedup allow you to gracefully scale down: you keep a complete table of successfully deduped blocks, plus a small fraction of other blocks, like e.g. smallest 1% of hashes, first blocks in files and final blocks in files. If you have a 12 MB (100 blocks) stretch that matches, you have 63% (1-1/e) chance of discovering one of these blocks as a match, and then use spatial locality to dedup adjacent blocks (duplicate blocks tend to come in bunches).
- jandrese 7y agoThen why does turning on dedup on ZFS suddenly make it chomp through 4GB of memory? Is there some tuneable you're supposed to be setting to make it less insane?