4 ms·
This approach can be very good for finding things like duplicate articles. It might not perform as well with blogspam, though. Rolling hash approaches can be v
by bunderbunder 12y ago
This approach can be very good for finding things like duplicate articles. It might not perform as well with blogspam, though. Rolling hash approaches can be very sensitive to "high-frequency" variation, so all a spammer would need to do to evade it is insert a reasonably high rate of spelling errors or word substitutions. That would have the effect of keeping the segment hashes themselves well-randomized (since even a single character difference should result in a completely different cryptographic hash), or prevent the rolling hash from selecting segment boundaries in a consistent manner.
One of the nice things about locality-sensitive hashing algorithms for less-than-pristine data is that you can instead tune it to construct the fingerprint from a larger number of smaller samples, which makes it more likely that you'll get at least some samples which represent all signal and no noise.
- 2510c39011c5 12y agowell, how do you define a spam? Is it any document that has been constructed around some key phrases or some urls that are in the spam database? If that's the case, why not just do a similar document evaluation see if the similarity of the document and some sample in the DB reaches certain threshold?
- ww520 12y agoGood point about introducing different spelling and word substitution. The original approach is for general data processing. If text document is the only kind of data, then add a pre-processing step to normalize the text. Remove all the stop words. Normalize words to their root to smooth out the different spellings and substitution. Even the Different word normalizations can be tried, Soundex, NYSIIS, or Metaphone.