3 ms·
Sounds like they don't use file data checksums... If they did, the dedup process for a multi tb drive would just be a few seconds, and the overhead would be so
by londons_explore 7y ago
Sounds like they don't use file data checksums...
If they did, the dedup process for a multi tb drive would just be a few seconds, and the overhead would be so small it could be run on the whole drive every minute all day long.
- sliken 7y agoBtrfs uses data and metadata checksums, those pre-existing checksums allow for checking for duplications without having to read the underlying files. So sounds like we are violently agreeing. Yes you can deduplicate 1TB quickly if you already have the data and metadata data checksums. ZFS (with dedupe on) checks the checksum of every written block against every existing block in the pool, thus the slow writes and being very ram intensive. Btrfs instead calculates data and metadata checksums for integrity then allows using them afterwards for deduplication. You could of course check for block level checksums for 100% of your storage. Or you could decide to dedupe only old files, only VMs, or only non-VMs. Or even a hybrid of block level for VMs, but file level for non-VMs.