3 ms·
To avoid the hash being entirely oblivious to bytes beyond a small block at the start, we could take additional samples of the file at exponentially increasing
by kazinator 5y ago
To avoid the hash being entirely oblivious to bytes beyond a small block at the start, we could take additional samples of the file at exponentially increasing offsets:
Hash the first kilobyte.
Hash the 16th kilobyte
Hash the 256th kilobyte
Hash the 4096th kilobyte (4M)
Hash the 65536th kilobyte (64M)
Hash the 1048576th kilobyte (1G)
Then 16G, 256G, ...
It might be worth including the last kilobyte. (I think I mentioned that.) If the file format has trailing metadata, that can help, especially if there is any sort of checksum in it.
- cb321 5y agoSure. I agree various sampling strategies are a good design, as long as they don't add up to much IO. Making that a "user parameter" to be customizable seems ideal. What you suggest may make a fine default - if adapted to varying file sizes somehow. E.g., that Nim dups program I mentioned elsewhere could just take a sequence of slices instead of only one..maybe just ignoring out of bounds slices. { And for size clusters where the size < N, screw it - just hash the whole thing, probably. ;-) }