4 ms·
Do you have data that is very obviously dedupeable? Or just a mix of things? A factor of three is not to be sniffed at.
by roygbiv2 2y ago
Do you have data that is very obviously dedupeable? Or just a mix of things? A factor of three is not to be sniffed at.
- watersb 2y agoThis archive was created as a dumping ground for all of my various computers, so a complete file by file dump of their system drives. While that obviously leads to duplicate data from files installed by operating systems, there's a lot of duplicate media libraries. (File-oriented dedupe might not be very effective for old iTunes collections, as iTunes stores metadata like how many times a song has been played in the same file as the audio data. So the hash value of a song will change every time it's played; it looks like a different file. ZFS block-level dedupe might still work ok here because nearly all of the blocks that comprise the song data will be identical.) Anyway. It's a huge pile of stuff, a holding area of data that should really be sorted into something small and rational. The application leads to a big payoff for deduplication.