4 ms·
While we are on this topic, does anyone know the current state of art for text duplication detection algos? I understand that Google used LSH but they must have
by boomzilla 13y ago
While we are on this topic, does anyone know the current state of art for text duplication detection algos? I understand that Google used LSH but they must have made a lot of progress since.
LSH: http://en.wikipedia.org/wiki/Locality-sensitive_hashing http://en.wikipedia.org/wiki/Locality-sensitive_hashing
- rossjudson 13y agoI'm not sure about the algorithm in use, but I what I hope is happening is that Google is now looking for the earliest publication of content when deduplicating. Most copycat sites have to copy their text from something existing, and if Google has already indexed that, they know that later versions of it are copies (and can presumably be knocked down in rank).