3 ms·
The OP’s paper reads: > Document Deduplication … We use the MinHash algorithm to compute 13-gram Jaccard similarities to determine which documents are near-dup
by threecheese 2y ago
The OP’s paper reads:
> Document Deduplication … We use the MinHash algorithm to compute 13-gram Jaccard similarities to determine which documents are near-duplicates of each other (Lee et al., 2021a). … We define two documents to be too similar when their Jaccard similarity exceeds 0.8, and randomly remove one of them.
The inline referenced [Lee et al](https://arxiv.org/pdf/2107.06499 https://arxiv.org/pdf/2107.06499) goes into detail, and notes that one of the algorithms is based on:
> 4.2 Approximate Matching with MinHash … perform approximate deduplication based on matching entire examples. This method, which we call NEARDUP, is a good complement to the exact substring matching, especially for web crawl text, as it handles the very common case of docu- ments being identical except for interspersed templated fields.
I’ve searched to my Peter Principle limit here, and was able to find [another implementation of MinHash](https://web.eecs.utk.edu/~jplank/plank/classes/cs494/494/notes/Min-Hash/ https://web.eecs.utk.edu/~jplank/plank/classes/cs494/494/not...) that used the first eight bytes of the hash, and so I am just going to assume that’s what they needed in order to meet their similarity KPI. Hopefully somebody here knows better/is smarter.