3 ms·
Well it's up to you. Generally you can look for duplicated blocks OR files, for any subset. So you could look at only files over 1 month. Or maybe any file o
by sliken 7y ago
Well it's up to you. Generally you can look for duplicated blocks OR files, for any subset. So you could look at only files over 1 month. Or maybe any file over 2GB you could look for block level deduplication and for anything under 2GB you could look for file level deduplication.
The utilities can be written in any language, there's several popular ones called dupremove, bedup, btrfs-dedupe, bees, and dduper.
Bedup specifically mentions "It integrates deeply with btrfs so that scans are incremental and low-impact."
For my use I'd likely ignore all files less than a few weeks old for deduplication purposes. Any virtual machine images I'd use block level deduplication and everything else I'd use file level deduplication.
Doesn't seem much different overhead wise than the normal scrub that linux software RAID, ZFS, and btrfs does anyways. I'm all for not storing old files twice, but would like to avoid the overhead of checking for deduplication on every write.
- londons_explore 7y agoSounds like they don't use file data checksums... If they did, the dedup process for a multi tb drive would just be a few seconds, and the overhead would be so small it could be run on the whole drive every minute all day long.
- sliken 7y agoBtrfs uses data and metadata checksums, those pre-existing checksums allow for checking for duplications without having to read the underlying files. So sounds like we are violently agreeing. Yes you can deduplicate 1TB quickly if you already have the data and metadata data checksums. ZFS (with dedupe on) checks the checksum of every written block against every existing block in the pool, thus the slow writes and being very ram intensive. Btrfs instead calculates data and metadata checksums for integrity then allows using them afterwards for deduplication. You could of course check for block level checksums for 100% of your storage. Or you could decide to dedupe only old files, only VMs, or only non-VMs. Or even a hybrid of block level for VMs, but file level for non-VMs.