Some form of locality-sensitive hashing such as MinHash or the algorithm presented in the linked article is generally good for general-purpose near-duplicate de

Some form of locality-sensitive hashing such as MinHash or the algorithm presented in the linked article is generally good for general-purpose near-duplicate detection. For when you've got a really big pool of documents to look through, simhash tends to be higher-performance because of the tiny fingerprints it uses. It also tends to force you into making stark decisions about precision vs. recall, though. Depending on your needs you might be able to make up for less-than-stellar precision by cleaning up false positives after the fact.