Locality-sensitive hashing works decently enough for finding duplicates/near-duplicates. Alternatively (and more simply if you already have a search index), Lu

Locality-sensitive hashing works decently enough for finding duplicates/near-duplicates. Alternatively (and more simply if you already have a search index), Lucene has a 'more like this' feature that also works pretty well in practice. It takes a document's tokens and submits them all to the search index in one query.