4 ms·
Maybe a good way to identify duplicates could be a comparison of word frequencies (or frequencies of 3 word sentences or something like that)? Perhaps that way
by Tichy 17y ago
Maybe a good way to identify duplicates could be a comparison of word frequencies (or frequencies of 3 word sentences or something like that)? Perhaps that way one could eliminate the "layout" and identify the core text.
I would like to create a library for that, if I could find the time :-/ Maybe then sites like HN would have an easier time to include the functionality.
Not sure if counting frequencies would be enough to identify the duplicates that have been filtered through translation. Interesting stuff.
- scott_s 17y agoNatural Language Toolkit for Python: http://www.nltk.org/Home http://www.nltk.org/Home It wouldn't be hard to use that to come up with a quick "fingerprint" of a page, consisting of word frequencies. The chances of two different pages having the same fingerprint would be exceedingly small.
- Tichy 17y agoI think there has to be some filtering out of irrelevant words, though. Rather than a unique fingerprint, there would have to be a measure on similarity. But I'll have a look at the NLTK, thanks for the link!
- RiderOfGiraffes 17y agoGiven that so many sites use some form of active content I should think that, unless designed very, very carefully, two separate loads of the same page could easily produce different "fingerprints." I've considered that and rejected it as too hard for a five minute trial, whereas the test against the title seems to work very well.