>Finding sections of the file where data can be different is not only potentially hard but depends on the type of data. So can't be made generic I agree, and a

>Finding sections of the file where data can be different is not only potentially hard but depends on the type of data. So can't be made generic I agree, and a cousin comment of mine suggests user-specifiable sampling which can always be a full sample in the "hard case". Often, once one has identified true duplicate candidates, a full compare is desirable to not trust any hash, and that full compare uses a lot of IO. That depends on the user, too. >Comparing files based on size is trivial and I presume people will always do it first before attempting to use a hash or even CRC. It is unclear the meow guys did this since they seemed to motivate hash performance based on dedup and they unlikely had IO > 10GB/s in 2018, but it's hard to know for sure.