I did some quick checks to investigate how size helps with deduplication. Size is a fast initial differentiator because scanning for file size only needs to re

I did some quick checks to investigate how size helps with deduplication. Size is a fast initial differentiator because scanning for file size only needs to read metadata, but how good does it differentiate? # all files find ~ -type f | wc -l 1098045 # files with a unique size find ~ -type f -printf '%s\n'|sort|uniq -c|grep ' 1 '|wc -l 48437 # most prevalent sizes (prevalence vs size) find ~ -type f -printf '%s\n'|sort |uniq -c |sort -nr|head 36972 0 12707 24 9480 22 8036 39 5025 12 4804 26 3879 6 3232 76 2946 38 2855 20 In my home dir, less than 5% of files have a unique size.