5 ms·
You actually don't need all that much authority to get away with ranking scraped content in Google. Despite their FUD, Google's duplicate content detection algo
by qeorge 17y ago
You actually don't need all that much authority to get away with ranking scraped content in Google. Despite their FUD, Google's duplicate content detection algorithm seems to be largely non-existent.
For example, check out the Google results for http://hackerne.ws http://hackerne.ws, which is a page-for-page duplicate of news.ycombinator:
http://www.google.com/q=site%3Ahackerne.ws http://www.google.com/q=site%3Ahackerne.ws
10,000 pages indexed, not a single word of original content.
Note: I know hackerne.ws is not trying to be spammy, and merely parked the domain improperly. If the owner is reading, all it would take is a simple 301 redirect to fix.
- blasdel 17y agoAll it would take to fix is for pg to make news.arc less shitty: actually check the HTTP/1.1 Host header, and respond with your own 301.
- dminor 17y agoOr include a rel=canonical link in the head.
- patio11 17y agorel=canonical won't work if the domain Google sees you on is different from the one specified as canonical. This is to prevent people capable of content injection from hijacking entire websites in a subtle manner.
- dminor 17y agoGoogle says otherwise: http://googlewebmastercentral.blogspot.com/2009/12/handling-legitimate-cross-domain.html http://googlewebmastercentral.blogspot.com/2009/12/handling-... No doubt they must use other indicators to ensure the authoritative source.
- merraksh 17y agoYour link didn't work for me, but this http://www.google.com/search?hl=en&site=q%3Dsite%3Ahackerne.ws&q=hackerne.ws&btnG=Search http://www.google.com/search?hl=en&site=q%3Dsite%3Ahacke... shows 255,000 results, of which hackerne.ws is the first, news.ycombinator.com is second :-| [edit: luckily, Google's duplicate content detection algorithm didn't work here...]
- slig 17y agoYou searched for the exact domain name of one site. It doesn't surprise me that it came in first.
- slig 17y agoThe fact tha the duplicated content is indexed doesn't mean the it ranks better that the original.
- qeorge 17y agoDefinitely true, but I became aware of this domain using Google to try to surface old threads. You get enough pages in the index, and it becomes a crapshoot on longtail searches. Here's an example: http://www.google.com/search?q=%22How+to+compensate+sweat+equity%3F%22 http://www.google.com/search?q=%22How+to+compensate+sweat+eq... In this case not only does the hackerne.ws page outrank the news.yc page, the news.yc has been pushed into the supplemental index.
- aditya 17y agoIt's a CNAME to news.ycombinator.com: lucidity% nslookup hackerne.ws Server: 192.168.0.1 Address: 192.168.0.1#53 Non-authoritative answer: Name: hackerne.ws Address: 174.132.225.106 lucidity% nslookup news.ycombinator.com Server: 192.168.0.1 Address: 192.168.0.1#53 Non-authoritative answer: Name: news.ycombinator.com Address: 174.132.225.106 lucidity%
- aditya 17y agoIt was a gift to HN: http://news.ycombinator.com/item?id=84039 http://news.ycombinator.com/item?id=84039
- qeorge 17y agoFound another one, that's just an IP: 174.132.225.106 [http://www.google.com/search?q=site%3A174.132.225.106 http://www.google.com/search?q=site%3A174.132.225.106] Another exact duplicate of HN, with 770,000 pages in Google's index.