3 ms·
You can do much better than that: you can use summaries or hashes to get a short version of each file and then compare the hashes for similarity. (Much like usi
by gwern 11y ago
You can do much better than that: you can use summaries or hashes to get a short version of each file and then compare the hashes for similarity. (Much like using perceptual hashes to find similar images.) A tool that does this is 'binsort'. It's O(N^2) IIRC, because of all the pairwise comparisons, but it's usable for up to a few hundred or thousand files.