4 ms·

There seem to be a number of sites that just poorly mirror content from StackOverflow. They usually have the question and then maybe some answers, but don't usu

There seem to be a number of sites that just poorly mirror content from StackOverflow. They usually have the question and then maybe some answers, but don't usually include comments to answers or even which answer was marked as correct. It's very frustrating when they come up above StackOverflow in Google searches. I end up slowing down a lot to make sure I don't click on those results. Sure I could just go straight to StackOverflow which I often do, but there are other good sources of programming knowledge on the internet that I want to be able to find too. These mirror sites are also frustrating because they are blatantly leaching off of the work of both the StackOverflow developers and the StackOverflow community. Kind of makes me wish Google did have a blacklist for these cases.

spolsky 16y ago
When we find out about these cases, we ask them to conform to our creative commons license agreement by linking back to stackoverflow.com, in hopes that ultimately this will teach Google that they are non-canonical and rank them lower than us.
- brc 16y ago
  That might be so, but right now there's about three of them and they clog up the search results with essentially the same answer but lower quality for the reasons mentioned above. So far I've not seen any that outrank stackoverflow, but it's only a matter of time before one does some blackhat and jumps for certain questions. You say that they link back, but in the cases I've seen, they link back with a slightly different Url form to the one you use. For example : http://stackoverflow.com/questions/1037925/recreate-stack-trace-with-line-numbers-from-user-bug-report-in-net http://stackoverflow.com/questions/1037925/recreate-stack-tr... this question appears on scraping sites (not going to name them for fear of feeding them traffic), but they link back with this Url: http://stackoverflow.com/questions/1037925 http://stackoverflow.com/questions/1037925 - which doesn't resolve back to the longer version. I know there's a canonical link on it (for the longer url) but I'm wondering if this strategy is working or not. Maybe it's time to start rejecting some bots crawling the site, or at least to slow them right down. It's a problem for Google and for SO, so someone from both places should probably be talking about it. Because in the end, SO starts looking spammy because the others are spammy and scrape the content, which is exactly the opposite case of how things should be. The spam sites are just cloning SO and shoving crappy ads everywhere, it's not high quality content. When you ask a question, you want 5 different answers from 5 different site, not the same answer spammed 5 times back at you.

spolsky 16y ago

When we find out about these cases, we ask them to conform to our creative commons license agreement by linking back to stackoverflow.com, in hopes that ultimately this will teach Google that they are non-canonical and rank them lower than us.

brc 16y ago
That might be so, but right now there's about three of them and they clog up the search results with essentially the same answer but lower quality for the reasons mentioned above. So far I've not seen any that outrank stackoverflow, but it's only a matter of time before one does some blackhat and jumps for certain questions. You say that they link back, but in the cases I've seen, they link back with a slightly different Url form to the one you use. For example : http://stackoverflow.com/questions/1037925/recreate-stack-trace-with-line-numbers-from-user-bug-report-in-net http://stackoverflow.com/questions/1037925/recreate-stack-tr... this question appears on scraping sites (not going to name them for fear of feeding them traffic), but they link back with this Url: http://stackoverflow.com/questions/1037925 http://stackoverflow.com/questions/1037925 - which doesn't resolve back to the longer version. I know there's a canonical link on it (for the longer url) but I'm wondering if this strategy is working or not. Maybe it's time to start rejecting some bots crawling the site, or at least to slow them right down. It's a problem for Google and for SO, so someone from both places should probably be talking about it. Because in the end, SO starts looking spammy because the others are spammy and scrape the content, which is exactly the opposite case of how things should be. The spam sites are just cloning SO and shoving crappy ads everywhere, it's not high quality content. When you ask a question, you want 5 different answers from 5 different site, not the same answer spammed 5 times back at you.