3 ms·
Natural Language Toolkit for Python: http://www.nltk.org/Home http://www.nltk.org/Home It wouldn't be hard to use that to come up with a quick "fingerprint" of
by scott_s 17y ago
Natural Language Toolkit for Python: http://www.nltk.org/Home http://www.nltk.org/Home
It wouldn't be hard to use that to come up with a quick "fingerprint" of a page, consisting of word frequencies. The chances of two different pages having the same fingerprint would be exceedingly small.
- Tichy 17y agoI think there has to be some filtering out of irrelevant words, though. Rather than a unique fingerprint, there would have to be a measure on similarity. But I'll have a look at the NLTK, thanks for the link!
- RiderOfGiraffes 17y agoGiven that so many sites use some form of active content I should think that, unless designed very, very carefully, two separate loads of the same page could easily produce different "fingerprints." I've considered that and rejected it as too hard for a five minute trial, whereas the test against the title seems to work very well.