3 ms·

What is the best way to deduplicate a corpus of documents

• marginalia_nu 3y ago
  If you mean in the sense of dealing with documents that are very similar but not binary identical, a locality sensitive hash would do the job.
• OnlyMortal 3y ago
  A Rabin finger printing algorithm and a hash of the data it generates. Reference count the hashes.