4 ms·
Maybe locality sensitive hashing or locality preserving hashing ? What sort of data are you storing?
by drdeca 10y ago
Maybe locality sensitive hashing or locality preserving hashing ?
What sort of data are you storing?
- passioncurious 10y agoThe content is pretty freeform-- one could be in html, could be in pdf, etc. To give a more concrete example, imagine having a million resumes (some in doc, some in pdf, some in html) and you want to dedupe them.
- yehosef 10y agoYou'll have to normalize them to text to make it work - the format is not important for the deduplication (meaning the same text in a word doc or pdf is really the same thing to you, even though at the byte level the files may be very different.)