3 ms·
There was some talk about improving ZFS deduplication considerably by not trying to be perfect. That is, try to deduplicate against a subset of the data, not ag
by magicalhippo 7y ago
There was some talk about improving ZFS deduplication considerably by not trying to be perfect. That is, try to deduplicate against a subset of the data, not against all the data.
This allows the dedup table (which stores the hashes of existing data to find identical blocks) to be limited in size to whatever you want, and especially prevent it from eating all the RAM. The tradeoff is that the smaller it is, the deduplication hit rate drops.
Here's a talk by Matt Ahrens about it, which also includes some other tricks to improve dedup performance:
https://www.youtube.com/watch?v=PYxFDBgxFS8 https://www.youtube.com/watch?v=PYxFDBgxFS8
http://open-zfs.org/w/images/8/8d/ZFS_dedup.pdf http://open-zfs.org/w/images/8/8d/ZFS_dedup.pdf