8 ms·
There has been a huge influx this year with the amount of sites that simply scrape SO and then have the exact content on their site. It's a pain, and there is n
by thejosh 4y ago
There has been a huge influx this year with the amount of sites that simply scrape SO and then have the exact content on their site. It's a pain, and there is no official way to remove them.
I thought that this was a massive nono from Googles side, has something changed?
- disruptiveink 4y agoI'd like to get actual confirmation of this, but my vague feeling is that, once upon a time, Google Search would get "updates", as in, actually deployed code that would change the rule of the game and most of the previous dirty tricks would become unusable, leading to people to go out and find out new ones. This changed with the Google "machine learning" days, where you no longer have humans at the helm laying down explicit rules, so no more "change the world" updates, you can only slightly nudge the parameters towards what you want, meaning the same old tricks keep being effective for far too long.
- nerdawson 4y agoThe “May Core Update” which recently rolled out impacted every site. A lot of updates are targeted at specific problems such as low quality product reviews but there are still broader updates taking place.
- dewey 4y ago> Google Search would get "updates", as in, actually deployed code that would change the rule of the game That's just what the scheduled "core update" days are now: https://developers.google.com/search/blog/2022/05/may-2022-core-update https://developers.google.com/search/blog/2022/05/may-2022-c...
- atwood22 4y agoI have no evidence of this, but the ad load on the returned results has gotten way higher. In theory, ranking sites that display Google ads higher would be a very easy knob for Google to turn to increase profit. The SO scrapers probably have Google ads on them, making them more profitable for Google.
- panarky 4y agoThis is a very old conspiracy theory that's been repeatedly debunked. https://www.searchenginejournal.com/ranking-factors/google-ads/ https://www.searchenginejournal.com/ranking-factors/google-a...
- chakkepolja 4y agoThat link is about AdWords spend by the site in question, and not about displaying AdSense ads on the site. Totally unrelated.
- hombre_fatal 4y agoI ran into so many Stack Overflow "mirrors" yesterday like this: https://www.anycodings.com/1questions/400836/swiftui-update-navigationview-after-deletion-ipad https://www.anycodings.com/1questions/400836/swiftui-update-... 10 years I gave up on a large project where I rehosted and organized dead Usenet forum content because Google's dupe-penalty detector was too good and too aggressive for content that you could barely find beyond a six-year-old cache hit where the origin website was long gone. Meanwhile these Stack Overflow scrapers are just `<html>{copy-and-paste}</html>` and the same domains are still alive despite years of cloning. Looks like it's time to boot my project back up.
- Nextgrid 4y agoTurning the knob one way explicitly might raise some anti-trust concerns, however the same motivation can be used to avoid turning the knob the other way and this can be done much more sneakily without leaving clear evidence - simply don't allocate budget/etc to projects that would turn the knob the other way and you're done.
- Quenhus 4y agoHere is my uBlock filter with hundreds of GitHub/StackOverflow copycats: https://github.com/quenhus/uBlock-Origin-dev-filter https://github.com/quenhus/uBlock-Origin-dev-filter It blocks copycats and hide them from multiple search engines. You may also use the list with uBlacklist.
- thejosh 4y agoThis is fantastic! This is exactly what I needed, thanks!
- Phlogi 4y agoThis even works on Firefox Nightly on Android. Thanks a lot!
- colordrops 4y agoWith these two pieces of data: * the identical text copied from stack overflow should be easily identifiable * volunteers put together a list of these sites themselves it should be obvious to Google apoligists that Google is either negligent or intentionally allowing these sites in their search. I'm sick of hearing about how "the world is different" and it's an "arms race" between spam sites and google. Bullshit.
- rightbyte 4y agoSO seem to have Yahoo ads, so I guess it is a no brainer for Google to rank sites they profit from over the content the lusers want.
- jiggawatts 4y agoThis is the real answer.
- remus 4y ago> the identical text copied from stack overflow should be easily identifiable Google starts matching content from SO => Spammers start tweaking the text slightly => google implements some expensive similarity score to down rank copy cat sites => spammers use more complex scrambling=> ... > volunteers put together a list of these sites themselves These lists only work because they're used by a tiny minority of people. If Google were to do this the spammers would start switching domains more quickly (or find some other workaround). I'm no Google apologist but I think you're underestimating how hard search ranking is when spammers are actively trying to game the system.
- maxwelldone 4y agoI've been using this uBO filter since someone recommended on a different thread and it's been great at removing those annoying sites from search results: https://github.com/quenhus/uBlock-Origin-dev-filter https://github.com/quenhus/uBlock-Origin-dev-filter
- burtekd 4y agoThe author acutally posted above your comment ;)
- skilled 4y agoI had this topic brought back to my mind yesterday as I was doing some research using the Ahrefs keyword tool. I do believe it would be possible to create a very large dataset of these copycat sites (using Ahrefs) to be used as a blacklist in various filters/extensions. But the crazy part is that, for example - Ahrefs says that StackOverflow has "Organic traffic" in the range of 22 million per month. A lot of these copycat sites, at least the ones I saw - have a traffic range anywhere from 10k to 500k per month. I mean, it's pretty insane just how well such sites can rank in Google, and you bet those copycats are making absolute bank from ads even if the majority of developers immediately close the site. There's a lot going on with Google Search these days, a lot of people are complaining that sites that scrape content can easily rank really well for long-tail keywords. One case in particular, a site will scrape Google to collect "featured snippets" and "people also ask" - then combined anywhere from 20 to 40 of these answers and publish them as a blog post. None of the words are changed, all questions/answers worded exactly the same. And Google puts these sites on page 1. What a joke.
- SheetPost 4y ago> bet those copycats are making absolute bank from ads even if the majority of developers immediately close the site I bet the majority of developers block ads
- helsinkiandrew 4y ago> I do believe it would be possible to create a very large dataset of these copycat sites Would they just move to creating and using new domains with the same content as soon as traffic to the old becomes drops? (What looks like the spammers in the original post are doing) But something does need to be done to these sites.
- pyinstallwoes 4y agoSimilar to YouTube search results. Lots of spam videos. No way to block a creator. Totally ruins it.
- Surfactant7 4y agoThere's a simple way around that. Nothing to install. Nothing to update. Just go to SO and use its search bar. It's actually quite good. I mean, you know that's where you'll want to find the answer anyway - not some random corporate webpage or ad-infested splog. Why not cut out the middle man? Only if that fails do I bother with Google.
- burnished 4y agoHuh, you know, you're right. I recently did that and it was fine. I think a lot of others formed their opinion (myself muchly included) about this from sites where the search bar was a joke played on people. Edit: let me upgrade that 'fine' to 'great', now that I think about it it was actually better than a google search which was not my previous experience.
- watchdogtimer 4y agoOr, if DuckDuckGo is your default search engine, you can append ' !so' to your search term.
- avereveard 4y agoGoogle index used to be fairly more competent at finding relevant issues for a query, especially if some words were synonyms of what found in the snippets at even loosely related
- thejosh 4y agoBecause it's not just SO I want answers from. It's blogs people post, social media such as Reddit, HN, etc. I've found some fantastic articles out there, yes SO is a fantastic resource but there is an entire internet out there :-)
- ofou 4y agojust switch to you.com
- minutillo 4y agoMy theory is that one of the inputs to Google's ranking algorithm is now "how much money would we make from this click?" A click to SO has a small number of ads which are obviously ads and easily ignored. A click to the average scrape-jacked SO page has dozens of ads using every dark pattern in the book to generate accidental clicks.
- fluidcruft 4y agoOne of the other commenters above made the claim that SO runs yahoo ads. If that's true then from a Google perspective, the click has either zero or negative money-making value. Maybe that means we should be searching in yahoo rather than google.
- xbar 4y agoThis reminds me of this one site that simply scraped all the open source code it could see and then produced AI-generated copies.
- endofreach 4y agohttps://news.ycombinator.com/item?id=29403947 https://news.ycombinator.com/item?id=29403947
- anonred 4y agoOther search engines allow you to block domains from showing up in the results. I’ve switched to Kagi out of frustration and honestly it’s as good or better than Google just because of that one feature.
- MaxDPS 4y agoI've been using uBlacklist and it works really well. It even lets me highlight specific websites so I have a better chance of seeing them if they are further down the list. https://iorate.github.io/ublacklist/docs https://iorate.github.io/ublacklist/docs
- phreack 4y agoIt took me a while for no good reason but I finally got an unofficial extension to add a "block" button to search results. It immediately improved my experience, I can't recommend it enough. No more Pinterest, SO clones, useless Quora spam, with very little work. I can't believe I didn't do it sooner.
- aaaaaaaaaaab 4y agoSome ex-Googlers say that someone ran an AB-test, and it turned out that per-search revenue was decreasing when these sites were blocked.