7 ms·
You really can't blame ZFS for needing ram for dedup. The raw math around looking up checksum just demands it. There are some other systems that can achieve fas
by mbrumlow 8y ago
You really can't blame ZFS for needing ram for dedup. The raw math around looking up checksum just demands it. There are some other systems that can achieve fast dedup, but instead of ram they would require a ton of super fast SSDs to store the tables.
There are some other types of dedup that don't guarantee 100% dedup, but then the dedup levels would very depending on data sets -- this may be fine for VMs and user data. They work by being using data locality, predicting which dedup table would needed to dedup the next N bytes. But it is not likely we are going to see this any time soon because the patents on this sort of dedup is fairly recent.
- LeoPanthera 8y ago> You really can't blame ZFS for needing ram for dedup. The raw math around looking up checksum just demands it. There are some other systems that can achieve fast dedup, but instead of ram they would require a ton of super fast SSDs to store the tables. This is only true for real-time, "live" dedup, which is an essential feature for almost nothing. btrfs' offline dedup (which, despite the name, can be done on a mounted filesystem) is far more rational. You fill the disk with data, and then dedup it later, at a schedule that works for you. No extra RAM needed.
- mbrumlow 8y ago> btrfs' offline dedup (which, despite the name, can be done on a mounted filesystem) is far more rational. You still have to process it, and you are still going to use a lot of ram -- and you are going to want to use a lot of ram once you get into larger data sets otherwise the process will take days.
- tatersolid 8y agoMost sane deduplication systems use a Bloom filter to avoid direct lookups to huge tables of content hashes. Not sure if ZFS does this or not, but even NTFS deduplication does.