3 ms·
My 4am thoughts on the matter... If a site has N copies associated with it, then you could compare that to the average number of copies associated with sites o
by alextgordon 16y ago
My 4am thoughts on the matter...
If a site has N copies associated with it, then you could compare that to the average number of copies associated with sites of that pagerank. If the difference between N and the average is high, then that's a likely spam site. Let's call this difference σ.
This gives us a problem though, because while σ is a good indicator of spamminess, it's not foolproof. A low pagerank site could have been copied from lots of times, which would unfairly earn it a high σ.
What we can do is calculate a σ', by inversely weighing each copy with the σ of the site on the other end. A copy of a site with a low sigma will increase σ', and a copy of a site with a high sigma will have less of an effect on σ'.
So while our high σ may have initially suggested the site to be spammy, all its copies are from high σ sites, which are counted less towards σ', leading to an overall verdict of not spammy. [I wonder what would happen if you iterated this process]
As for your example, the smaller site wouldn't be penalized, because N would be low. If it were say, a full wikipedia mirror, then it would be penalized. Wikipedia would not be penalized, since it has an ungodly amount of pagerank. There's also no bias against smaller sites, since σ is calculated relative to pagerank.
What have I missed?