Please consider enhanced duplicate detection as outlined and discussed here: http://news.ycombinator.com/item?id=1012215 http://news.ycombinator.com/item?id=10

Please consider enhanced duplicate detection as outlined and discussed here: http://news.ycombinator.com/item?id=1012215 http://news.ycombinator.com/item?id=1012215 Simply compare the "title" field of the referenced page. If it matches, perhaps compare the domain name, ignoring any "www". Or perhaps not bother. The number of occasions the title matches when the page is different should be sufficiently small.