6 ms·
I’ve tried dedup out, and even with a large powerful box with a LOT of duplicate files (multi-TB repositories of media files which get duplicated several times
by lazide 5y ago
I’ve tried dedup out, and even with a large powerful box with a LOT of duplicate files (multi-TB repositories of media files which get duplicated several times due to coarse snapshotting from other less fancy systems), I get near zero deduplication. I think it was literally low single digits percents.
ZFS dedup is block based, and actual block size varies depending on data feed rate for most workloads (zfs queues up async writes and merges them), so in practice once a file gets some non-zero block offset somewhere which happens all the time, even identical files don’t dedup.
- Wowfunhappy 5y agoWow, that’s worse than I realized! Honestly, this makes me wonder whether the feature should even exist in ZFS. Given the enormous hardware requirements and minimal savings... well, I’d be curious to hear if anyone has ever found a real use case.
- FullyFunctional 5y agoDoes anyone actually use dedup? I think even the OpenZFS documentation says compression is more useful in practice. If at all, dedup should be an offline feature, to be run as scheduled by the operator. My setup tries to get the absolute highest bandwidth and uses NVMe sticks in a stripe (I get my redundancy elsewhere), no compression, no dedup and yet can only hit ~ 3.5 GB/s reads (TrueNAS Core, EPYC 7443P, Samsung 980PRO, 256 GiB). I hope TrueNAS SCALE will perform better.
- bombcar 5y agoIt would be nice if ZFS was able to combine dedup and compression - basically be able to notice that a block/file/datastream was similar/identical to another one, and do compression along with a pointer ...
- willis936 5y agoZFS can have both features enabled at once. Though there is no clean way to disable either. Compression can be removed from files by rewriting them, but removing deduplication requires copying over all data to a fresh pool.
- nightfly 5y agoYou should be able to remove the deduplication data by zfs send/recv within the same pool too, I think
- willis936 5y agoNot according to people who know more about ZFS than me. https://www.truenas.com/community/threads/zfs-dedup-disable-cleanup.98917/ https://www.truenas.com/community/threads/zfs-dedup-disable-...
- nightfly 5y agoSo at a pool level you might not be able to turn it off once it's turned on, but you can also turn off deduplication per file system, including in properties you set when receiving a stream. I wasn't confident this would work, but a test proved it can. (chicken_test/dedup_source had deduplication enabled and 16 copies of the same 100MiB file) chicken:~# zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT chicken_test 15G 144M 14.9G - - 0% 0% 16.00x ONLINE - chicken:~# zfs send chicken_test/dedup_source@send | zfs recv -o dedup=off chicken_test/nodedup_dest chicken:~# zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT chicken_test 15G 2.29G 12.7G - - 0% 15% 16.00x ONLINE - chicken:~# zfs get dedup chicken_test/nodedup_dest NAME PROPERTY VALUE SOURCE chicken_test/nodedup_dest dedup off local chicken:~# zfs destroy -r chicken_test/dedup_source chicken:~# zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT chicken_test 15G 2.29G 12.7G - - 0% 15% 1.00x ONLINE -
- 5y ago
- lazide 5y agopractically speaking the tradeoffs to make that work are unlikely to make you or anyone else happy except in some VERY specific workloads.
- watersb 5y agoMy first ever large (> 4TB) ZFS pool is still stuck with dedup. It's a backup server, gets about 2x with deduplication. At the time, it was the difference between slow and impossible: I couldn't afford another 2x of disks. These days, the pool could fit on a portable SSD that would fit in my pocket. Careful, file-based dedup on top of ZFS might be more effective. Small changes to single, large files see some advantage with block based deduplication. You see this in collections disk images for virtual machines. You might see that in database applications, depending on log structure. I don't know, I don't have that experience. For most of us, file-based deduplication might work out better, and is almost certainly easier to understand. You can come with a mental model of what you're working with, dealing with successive collections of files. Even though files are just another abstraction over blocks, it's an abstraction that leaks less without the deduplication. I haven't used a combination of encryption and deduplication. That was Really Hard for ZFS to implement, and I'm not sure how meaningful such a combination is in practice.
- justinclift 5y ago> no compression, no dedup and yet can only hit ~ 3.5 GB/s reads (TrueNAS Core, EPYC 7443P, Samsung 980PRO, 256 GiB) Hmmm, that 3.5GB/s sound low. From rough memory of doing initial storage benchmarking of our "new" Hetzner dedicated boxes a few months ago (AX51-NVMe, https://www.hetzner.com/dedicated-rootserver/ax51-nvme https://www.hetzner.com/dedicated-rootserver/ax51-nvme), they were giving about 10GB/s with mirrored NVMe drives. Just logged into one of those boxes now, and it's running 2x 1TB Samsung PM9A1 drives (https://semiconductor.samsung.com/ssd/pc-ssd/pm9a1/ https://semiconductor.samsung.com/ssd/pc-ssd/pm9a1/), compression is on (lz4), and dedup is off. (Didn't do any real tuning at the time, as these specs were already far in excess of what's needed for these servers.) Would enabling lz4 compression be useful for your use case?
- FullyFunctional 5y agoThanks for the insight. I'll can try that. Note however that reading from the raw device on the same hardware & OS I hit 5+ GiB/s (IIRC).
- yjftsjthsd-h 5y ago> Honestly, this makes me wonder whether the feature should even exist in ZFS. My understanding is that the OpenZFS project devs feels that the answer is an emphatic no, it should not exist, but they're committed to backwards compatibility so they won't drop it. (Take with a grain of salt; that's an old memory and I can't seem to find a source in 30s of searching.)
- spullara 5y agoWow, that is a very naive dedup algorithm.
- lazide 5y agoWithout restricting pretty heavily how you can interact with files or causing severe bottlenecks it's the best that can be done probably, since the FS API doesn't provide any real guarantees about what data WILL be written later, or how much of it, etc. So it has to figure things out as it goes with minimal performance impact.
- mustache_kimono 5y agoHas anyone done any better at the FS layer?
- magicalhippo 5y agoWhile regular dedup is only a win for highly specific workload, the file-based deduplication[1][2] which is in the works seems like it can have some potential. They discussed it, along with some options for a background-scanning dedup service (trying to find potential files to dedup), in the February leadership meeting[3]. [1]: https://openzfs.org/wiki/OpenZFS_Developer_Summit_2020_talks#File_Cloning_with_Block_Reference_Table_.28Pawel_Dawidek.29 https://openzfs.org/wiki/OpenZFS_Developer_Summit_2020_talks... [2]: https://youtu.be/hYBgoaQC-vo https://youtu.be/hYBgoaQC-vo [3]: https://www.youtube.com/watch?v=hij7PGGjevc https://www.youtube.com/watch?v=hij7PGGjevc
- oofbey 5y agoThis sounds cool. Current block dedup really seems like a simple (naive) v1 compared to what should be possible.
- mlok 5y agoZFS Dedup has been wonderful for me : dedupratio = 7.05x (144 GB stored on a 25 GB volume, and still 1.3 GB left free). I use it for backups of versions of the same folders and files slowly evolving over a long period of time ( > 15 years) that gives a lot of duplication, of course. (I could also use compression on top of it)
- infogulch 5y agoBackups are the 99% case for duplicate files, but aren't snapshots a better replacement in every way? Snapshots are already deduplicated as soon as you take them, plus they're instant. Maybe if your backups are coming from a non-zfs system, but you could probably convert normal backups into snapshots without too much trouble. Why is dedup even present when the primary use case (backups) is better served in every way by snapshots?
- toast0 5y agoDedup (if it worked like it might have!) could solve the backup use case without needing to dictate your workflow. In theory, it could also really help with virtualized disk workloads where there may be a lot of duplicated data from the base OS, but you can't use a snapshot (easily) because windows won't run from a zfs filesystem. You could maybe do snapshotting on zfs volumes, but that's not as flexible as a dedupe that worked as imagined. Personally, I think online dedupe ends up being too expensive in memory and computation and ends up missing things because of divergent block sizes or offsets as another poster mentioned. ZFS doesn't support an offline dedupe, but I think btrfs does. That might be more interesting. It's still expensive to find duplicates, but it's possible, and it'd be neat to be able to rewrite the metadata to refer to a single copy and free some space, maybe.