> You should implement a "unique" webpage result fingerprint in order to avoid showing duplicates (well, I suppose it's easier said than done..). It is, mainly

> You should implement a "unique" webpage result fingerprint in order to avoid showing duplicates (well, I suppose it's easier said than done..). It is, mainly because people have very different ideas what "unique" actually is. At one time I had add the host of a page to their hash because people complained that if we hit the same content at two different hosts we filtered one out, which wasn't okay for political reasons. The two first results in your example are "fun". That page returns different content with each call ... yay.