3 ms·
Interesting but I'm not sure if it's any better than digram or trigram frequency. The real difficulty of course is the k-nearest-neighbors problem when you are
by cameldrv 18y ago
Interesting but I'm not sure if it's any better than digram or trigram frequency. The real difficulty of course is the k-nearest-neighbors problem when you are trying to search for the duplicate.
- gaika 18y agoIt is more efficient, as the distance between the duplicates in their case is a lot shorter (because effectively they only compare the important part of the page). That lets you use simple algorithms that are a lot faster. This is still not foolproof: some sites have an about header with lots of text or blogs that have a blurb before and after a quote.
- stcredzero 18y agoYou can think of this as just changing the granularity of your tokens. The set of possible tokens becomes huge and the number of them per page decreases at the same time, so of course this is going to make looking for approximate matches much easier.