6 ms·

Ask HN: Has anyone else noticed Stack Overflow clones in Google search results?

71 points by rchrd2 11y ago

Has anyone else noticed Stack Overflow clones in Google search results? They come up frequently for me. I can't help but wonder who's behind these. It can't be hurting Stack Overflow's SEO. So far I have saved 5 different domains, and it looks like 2 have vanished. - [dead] http://www.codeitive.com/0izVUjjXVP/selective-foreign-key-usage-in-django-maybe-with-limitchoicesto-argument.html - [dead] http://www.codedisqus.com/0QmqWVgjgg/hide-label-in-django-admin-fieldset-readonly-field.html - http://w3facility.org/question/image-servingurl-and-google-storage-blobkey-not-working-on-development-server/ - http://goobbe.com/questions/3109325/how-can-i-disable-a-third-party-api-when-executing-django-unit-tests - http://www.ciiycode.com/0HyN6eQxgjXP/django-admin-inline-popups I actually wrote Stack Overflow support about this in April 2015, but so far nothing has changed. Here's the thread: Me: "Hello, There a lots of spam results on Google. As a web developer, I am frequently googling for how to resolve some programming issue. Often, I get these spoofing sites that link to stack overflow. I would suggest adding a stricter robots.txt or perhaps blocking some of these bots that are scraping your site. Here is an example. http://www.ciiycode.com/0HyN6eQxgjXP/django-admin-inline-popups Thank you." Response: "Hello, Thank you for reporting this content. I've passed the information along to the person at our company who handles such issues. It's the diligence of users like you that helps us stay valuable! Please note, bringing these sites into compliance (or getting them to no longer serve our content) is often a long and arduous process. You may not see immediate results. However, rest assured that we're working on it. Thank you again, Stack Exchange Team" Thoughts?

nmbdesign 11y ago
Have been noticing this for the last few months too, very weird.
kjhughes 11y ago
See "A site (or scraper) is copying content from Stack Exchange. What do I do?" http://meta.stackexchange.com/q/200177/234215 http://meta.stackexchange.com/q/200177/234215
jrockway 11y ago
Stack Exchange makes all the content available as a downloadable file, so they must be expecting clones. https://archive.org/details/stackexchange https://archive.org/details/stackexchange They then have a bunch of licensing terms if you reuse the content, which must be what they're referring to by "bringing these sites into compliance is often a long and arduous process".
- ivanca 11y ago
  They even have a tool/site to query the data: https://data.stackexchange.com/stackoverflow/query/new https://data.stackexchange.com/stackoverflow/query/new
plorkyeran 11y ago
They've been around for years. A while back a few were sometimes beating SO in google results, but google eventually fixed that. Everything on SO is cc-by-sa licensed, so as long as the source is attributed (and it is on some of the sites), it's legal. The motivation is simple: the clone spam sites have ads on them. They're not actively trying to attack SO; they're just leeching value.
- rchrd2 11y ago
  I guess it's a win-win situation. Stack Overflow's ranking goes up, because of all the sites referencing it, and the spammers make money. But as a consequence, the users have to sift through spam results.
  - ibejoeb 11y ago
    I don't think so. Those sites redisplay SO content, but they're not linking to SO. They're actually trying to out-rank them, up their own traffic, and drive up their ad revenue.
    - rhizome 11y ago
      Sometimes they link back, sometimes they don't.
volaski 11y ago
I don't understand why StackOverflow allows this, yeah creative commons is cool and all but it IS NOT COOL for actual users. I get so annoyed every time i search for something on Google and it leads to an its clone site. It's not like StackOverflow has better search than Google (which is ridiculous). I still have to search on Google if I want quality search result instead of StackOverflow. With more power comes more responsibility. StackOverflow's policy feels too irresponsible in this regard
- rhinoceraptor 11y ago
  Just append your query text with 'site:stackoverflow.com'.
  - fezz 11y ago
    That or use the SO docset in Dash or StackStash https://itunes.apple.com/en/app/stackstash-stackoverflow-offline/id602112351?l=en&mt=8 https://itunes.apple.com/en/app/stackstash-stackoverflow-off... https://news.ycombinator.com/item?id=7932477 https://news.ycombinator.com/item?id=7932477
- JasonPunyon 11y ago
  We use that license because it protects the content from us. No matter who comes along to run Stack Overflow in the future, Stack Overflow can't do something like put up a paywall and lock it up. Someone else'll just be able to host a copy.
  - msie 11y ago
    Yes, look at what happened to imdb.
  - volaski 11y ago
    As an end user, I don't care about that at all. If StackOverflow does something like put up a paywall and lock it up, then it will die off and some other site will arise that will replace it, just like some other sites that came before StackOverflow which put up a paywall and faded away. Also, StackOverflow can always change the policy if they want (which probably won't happen for the reason I mentioned), so the license as an excuse doesn't really make sense to me. Especially when it comes at a cost of horrible user experience. Lastly it doesn't seem like StackOverflow is doing much to improve search on the site itself and that's what makes this even worse. I wouldn't be complaining if I could find more relevant StackOverflow results on StackOverflow than searching on Google. How is it that I can find more relevant results on a generic search engine than the site where the contents came from?
fiatjaf 11y ago
I don't understand why Google can't figure it out and remove these clones. They can do much harder things. Why couldn't them outrank sites based on equal text content or -- much better -- huge presence of ads.
- rhizome 11y ago
  They have made inroads in the past, but lately copycats have been cropping up in results (iswwwup.com is one I've been seeing a lot) again. I imagine there's a ranking algo update in the future that might fine-tune this more. To be sure, it's going to be an arms race, since the only purpose of these sites is adsense siphoning.
- TeMPOraL 11y ago
  I suppose it's mostly a social/legal problem. Google could get rid of 90% of those scrappers by basically hardcoding a preference for StackOverflow if it has the same content than another site. Same with sites like Wikipedia. But then obviously people will call foul play (probably scrappers and SEO people will be the loudest to complain). So whatever solution Google makes has to be general enough people won't call it unfair, and that makes the problem much more difficult.
Animats 11y ago
Google doesn't have a good way to establish provenance, and has trouble distinguishing copies from originals. It's a common complaint of blog operators that some bigger blog copied their stuff and got a higher ranking on Google. Google could check when it saw something, but that won't work against fast scrapers. For that, you need trusted timestamps. One solution to this would be to have a few time-stamping services. You send in a string, probably a hash, and it adds a timestamp, signs it, and sends back a signed result. Then provide a WordPress plug-in to use this service, hashing and time-stamping each blog entry, and putting the result in the HTML in some standard way. (Perhaps <span signed-provenance-timestamp-hash="xxxxx"> blog entry </span>). A few mutually mistrustful services for that would help; blogs with serious forgery problems could use multiple time-stamping services. Search engines then need to look at timestamps as a rating indicator. If two results are very similar, the earliest one wins.
- Osmium 11y ago
  Since establishing provenance is such a big problem for Google, perhaps it might be a good idea for Google to offer a time-stamping service itself?
  - dsjoerg 11y ago
    It's not directly a problem for Google — it's primarily a problem for sites that create original content.
    - ezequiel-garzon 11y ago
      Not quite. The quality as a search engine could improve dramatically by letting true authors let Google know the content is coming before it's been available anywhere on the web.
    - what_ever 11y ago
      Well, I think Google would prefer to send the traffic to StackOverflow instead of it's clone as it's better source?
    - 11y ago
onion2k 11y ago
It's not just StackEnchange. I noticed the other day there's a Twitter account and website called "@explodingAds" that tweets HN user comments and mirrors them on it's website. I imagine taking content verbatim is just a quick way to build a corpus of search indexed pages that generate page views and ad impressions.
wyclif 11y ago
There's also a lot of Trello clones.
jeffmould 11y ago
Sort of related, but in the last year it seems to me that Google's results in general have been increasingly getting worse. Now it seems that at least two or three spam results are always present in the first page of results. Most of these results also happen to be duplicate content of bigger sites though. I have tried reporting on numerous occasions to Google, but just usual "we will investigate" response and never a word back.
- Bjartr 11y ago
  In the cases where the information you're looking for doesn't need to be recent (in my case exercises for a particular sport) you can change the date range of your search to e.g. only include results from prior to 2005. Obviously this isn't an option for recent information, but I've found it useful on occasion to filter out blogspam.
- deleted 11y ago
  [deleted]
dredmorbius 11y ago
This is common across numerous contexts. I've found what appear to be bots Tweeting my reddit and HN posts (I don't mind), several Reddit clones of various levels of sniffitude, Google+ content harvesting, and some Diaspora syndication (to be expected), again, of various levels of sniffitude. To the extent that this simply distributes data around, doesn't claim it for its own, and credits source, I'm OK with this. Better even if it follows site-specific licensing. Among my visions for the Web would be content syndication where such schemes would actually directly benefit authors and creators, regardless of where their content is served.
Phil_Latio 11y ago
When I look at w3facility.org, it seems like Googles algorithms do not properly handle the case when a scrape-site provides a source-link to the original content. Google recommends the latter to protect against duplicate content penalties when you use some external content to enrich your site (for example a short section from Wikipedia, imdb actor info, etc).
douche 11y ago
I see this happen a lot also with the MSDN forums. The interesting thing there is that some of the mirror sites are still carrying topics that have been deleted or otherwise disappeared from the real MSDN forums. More than once, in the obscure subset of Microsoft tooling that I work in, the only hits that are still alive are on somewhat suspicious looking .ru sites, so in some sense, I am glad these sites do exist - otherwise, I'd be completely SOL trying to figure out why the badly-documented API I'm relying on is barfing up an opaque HRESULT.
aaron695 11y ago
> I would suggest adding a stricter robots.txt or perhaps blocking some of these bots that are scraping your site. The only people who care about robots.txt are some of the big companies. Even Baidu ignores it (As they can, it's purely there as etiquette) Blocking bots is hard.
inguinalhernia 11y ago
Scraper spam exists for virtually every website that is even somewhat popular, there is almost nothing site owners can do about it but rely on Google, Bing, DuckDuckGo, etc, to figure out the originator. Google should be able to figure it out and rank the true origin source appropriately, and usually Google is pretty effective at that. But sometimes they're not. Perhaps what I find ironic is that many of the user "answers" on StackOverflow are basically clone spam themselves, copy/pasted from other websites by some user of the site, usually without sourcing the origin. I have personally found my own unique solutions and code copied verbatim and pasted to answers on the StackExchange network multiple times, outranking my original work without a reference to the origination of course, and I'm sure others have experienced something similar. Perhaps that's what you get with a user generated site, maybe Wikipedia experiences something similar. Related, some of you may recall a few years back, that StackOverflow basically complained to Google in a public fashion about not ranking well enough and got a boost from them, whereas obviously the average Joe and an average website has no such option nor recourse. Here was the discussion on HN: https://news.ycombinator.com/item?id=2152286 https://news.ycombinator.com/item?id=2152286
ConceptJunkie 11y ago
I haven't seen this yet, and I do searches pretty frequently that result in StackOverflow.com hits... and I make a point of choosing them first, because they are usually the best results. However, a few days ago, I did see a single hit for expertsexchange.com for first time in years, at least that I noticed. Back when Google used to let you blacklist domains from your search results, before StackExchange was ever around, or when it was still very new, I used to block them, and eventually I'd assumed they'd gone away, but maybe not. I hope StackOverflow and/or Google are able to do something to put the kibbosh on this kind of thing, because despite the complains SO is still a huge and valuable resource.

nostrademons 11y ago

Google offers this service already: http://www.labnol.org/internet/fat-pings-for-content-scrapers/20400/ http://www.labnol.org/internet/fat-pings-for-content-scraper... https://en.wikipedia.org/wiki/PubSubHubbub https://en.wikipedia.org/wiki/PubSubHubbub https://pubsubhubbub.appspot.com/ https://pubsubhubbub.appspot.com/ Most of the big hosted publishing platforms like WordPress and Blogger already use it, but it's pretty common for sites that built their own codebase not to. (This was one of my interests at Google, and I had both a 20% project [unreleased] and a real project [Google Authorship] that were devoted to algorithmically detecting copied content and providing reliable attributions. Ultimate though, FatPing/PubSubHubbub [done by another team] was a much more robust way of doing this, with the downside that it's on the webmaster to implement it.)

ezequiel-garzon 11y ago
Thanks for sharing this. I stand corrected (by a Googler, no less!). I wish Google pushed for greater adoption, then, as it does with Page Speed, SEO advice, etc.
sltkr 11y ago
That's not a timestamping service; that's just a way to quickly notify Google when you add a page to your blog. What if your blog doesn't use Pubsubhubbub, but mine does, and I copy your blog post? Google will see my post first, but that doesn't prove I wrote it, so Google cannot use that to make ranking decisions.
- thaumasiotes 11y ago
  I don't see how pubsubhubbub differs from the originally proposed timestamping service. For any timestamping service, it will still be the case that if you don't use it, and somebody else scrapes you and does use it, their timestamp will be better than yours.
  - nitrogen 11y ago
    There can be more than one timestamping service.
    - nostrademons 11y ago
      This is not an advantage. Imagine a scenario where a rogue timestamping service performs just well enough to get a significant userbase. They then alter their service such that instead of simply returning the timestamp, they put the content up on the Internet, but assign themself an earlier signed timestamp before returning a timestamp to the original requestor. They would instantly replace all of their customers in Google results, because Google would think that they were the originators of the content. They'd then be free to do all sorts of mischief with that influx of traffic: show ads, install malware, phish addresses, etc.

codezero 11y ago

Isn't part of the core of pagerank that certain domains are more trustworthy than others? Given that this is supposedly true, wouldn't the same content on stackoverflow out rank a random new domain with the same content

equalarrow 11y ago
Exactly. In the SEO world, the common wisdom is you will get dinged in ranking or even deindexed if you publish duplicate content. Period. There is an entire sub-industry in SEO that's all about content spinning. I'm always kinda amazed that these sites show up and I usually get them for about 50%+ of my tech searches. What I think ultimately happens is Google is notified and these sites are eventually deindexed. But this still leads me to really question the whole duplicate content theory sometimes. If Google was so tight about this, these sites would never show up in the first place. It could also be that the tech terms searched for are not as competitive as, say, medical or diet or celeb terms. Maybe Google's just grabbing at any kind of relevancy at this point, duplicate or not.
- codezero 11y ago
  This is really surprising. Given that they have so much data, there should at least be several thousand domains that they treat with some special regard when content that replicates them shows up. Then again, I'm sure that there are plenty of top 1000 domains that would use this to their advantage too, oh well.
andrew-lucker 11y ago
pagerank works both at the domain and page level, originally with the emphasis on page level(hence pagerank not domainrank). While it would be hard for these sites to match the cumulative reach of SO, they will have an easier time getting specific pages to rank highly. This can also be abused with a system where most pages link to the .1% of pages that should be emphasized. In this fashion, smaller sites can throw their weight around and get specific pages to rank more highly than bigger sites. Trying to solve this issue on a purely technical level turns out to be a lot more complicated than it would seem. It is made much worse by how damaging false-positives are for a search engine (Google censored me!). So the result is that this often only gets resolved by manual actions rather than automatic penalties.
- wodenokoto 11y ago
  Actually it's pagerank for Larry Page
  - tripzilch 11y ago
    hence his name is not Larry Domain.
  - WaltPurvis 11y ago
    Actually it's not. (In case anyone might think you were being serious.)