13 ms·
Finding sections of the file where data can be different is not only potentially hard but depends on the type of data. So can't be made generic. Comparing file
by DeathArrow 5y ago
Finding sections of the file where data can be different is not only potentially hard but depends on the type of data. So can't be made generic.
Comparing files based on size is trivial and I presume people will always do it first before attempting to use a hash or even CRC.
As you say, single disk read speeds are slow but there is a high chance they will improve over time.
PCIe 6.0 x16 has a bandwidth of 128 GB/s. It would be theoretically possible to reach that aggregate speed in raid 0.
- cb321 5y ago>Finding sections of the file where data can be different is not only potentially hard but depends on the type of data. So can't be made generic I agree, and a cousin comment of mine suggests user-specifiable sampling which can always be a full sample in the "hard case". Often, once one has identified true duplicate candidates, a full compare is desirable to not trust any hash, and that full compare uses a lot of IO. That depends on the user, too. >Comparing files based on size is trivial and I presume people will always do it first before attempting to use a hash or even CRC. It is unclear the meow guys did this since they seemed to motivate hash performance based on dedup and they unlikely had IO > 10GB/s in 2018, but it's hard to know for sure.