5 ms·

It’s a general problem. In the past I have seen great HN threads sharing exclude lists etc to avoid the scrapers. Google could, of course, track a few thousan

It’s a general problem. In the past I have seen great HN threads sharing exclude lists etc to avoid the scrapers. Google could, of course, track a few thousand canonical sources so that just python and SO and a few others don’t get beaten to the top in their specific expert areas, but it doesn’t genrtalize. Google has no real algorithmic idea who is copying who. And then it hits me: the proper documentation for things are on pages without ads! Perhaps that’s the signal google needs to start weighing heaviest…? ;)

menacingly 3y ago
SO isn't a canonical source, they're another content farm using nofollow links to the content they appropriate to maintain their search edge. Its entire purpose is to inject itself as a search middleman. Even worse, I find myself landing there from google results to a question they had closed as not worthy of the site, but worthy enough still to keep up for the search juice.
- layer8 3y ago
  > I find myself landing there from google results to a question they had closed as not worthy of the site Well, you did google for the question and not for the answer. ;)
  - gobdovan 3y ago
    True, and it was even more accurate a few years ago. Now Google moved towards understanding queries and giving context-specific answers rather than a strictly keyword based engine. You were considered a bad googler if you asked it a question, now it's the norm.
BugsJustFindMe 3y ago
> Google has no real algorithmic idea who is copying who. This is an irrelevant distraction, because Google's literal army of nearly 200,000 full time employees, a large number of whom are Python developers, do know. Geeksforgeeks, w3schools, programiz, tutorialspoint, and python-reference.readthedocs, which all rank higher than the official documentation, are not flash-in-the-pan sites. They've been polluting the search results for literally years. Trivial manual processes at Google scale would be 1000% fine and effective. Google could, if it gave half a shit, give every employee a Chrome browser extension that lets them manually vote on the reasonability of site rankings within known problem genres for their own searches.
- soraminazuki 3y ago
  I have the feeling that this very page would be downranked very fast if such an extension existed.
- rg111 3y ago
  Honest question: haven't people developed something akin to mental-ad-masks when it comes to Programiz, G4G, TP, etc.? Like I never ever ever click on a web ad, and can't even remember the brand it was about 3 seconds after leaving the page, whenever I see G4G, TP, etc., I never ever ever click them. At this point, I think this doesn't even reach my brain, but is handled by my spinal chord.
  - MathMonkeyMan 3y ago
    It does reach your brain, and it does affect you, just like a fog of gnats you've grown accustomed to still affects you. For what it's worth, the ads aren't for you. They're for anybody or anything that might click them, knowingly or not, and that's not you.
deepserket 3y ago
Isn't boosting websites without ads against their interest?
- wood_spirit 3y ago
  This is what I was hoping all the replies would be about :) Is monetising the web through advertising the reason these scraper sites exist? And removing that would make scraping sites an expense with costs rather than profits? Perhaps google could do this. Ads on third party sites are probably a small part of their income compared to sponsored ads in search results, so they could be sacrificed and they’d take down their competitors who rely more on the website ads because they don’t have the search? However, they have no need to compete on search quality at this time. If another player in the search and browser wars started downweighimg ad heavy sites then this might make google have to reluctantly follow suit. But you can imagine the FUD campaigns to paint such an actor as killing the internet and being the bad guy etc…

menacingly 3y ago

SO isn't a canonical source, they're another content farm using nofollow links to the content they appropriate to maintain their search edge. Its entire purpose is to inject itself as a search middleman. Even worse, I find myself landing there from google results to a question they had closed as not worthy of the site, but worthy enough still to keep up for the search juice.

layer8 3y ago
> I find myself landing there from google results to a question they had closed as not worthy of the site Well, you did google for the question and not for the answer. ;)
- gobdovan 3y ago
  True, and it was even more accurate a few years ago. Now Google moved towards understanding queries and giving context-specific answers rather than a strictly keyword based engine. You were considered a bad googler if you asked it a question, now it's the norm.

BugsJustFindMe 3y ago

> Google has no real algorithmic idea who is copying who. This is an irrelevant distraction, because Google's literal army of nearly 200,000 full time employees, a large number of whom are Python developers, do know. Geeksforgeeks, w3schools, programiz, tutorialspoint, and python-reference.readthedocs, which all rank higher than the official documentation, are not flash-in-the-pan sites. They've been polluting the search results for literally years. Trivial manual processes at Google scale would be 1000% fine and effective. Google could, if it gave half a shit, give every employee a Chrome browser extension that lets them manually vote on the reasonability of site rankings within known problem genres for their own searches.

soraminazuki 3y ago
I have the feeling that this very page would be downranked very fast if such an extension existed.
rg111 3y ago
Honest question: haven't people developed something akin to mental-ad-masks when it comes to Programiz, G4G, TP, etc.? Like I never ever ever click on a web ad, and can't even remember the brand it was about 3 seconds after leaving the page, whenever I see G4G, TP, etc., I never ever ever click them. At this point, I think this doesn't even reach my brain, but is handled by my spinal chord.
- MathMonkeyMan 3y ago
  It does reach your brain, and it does affect you, just like a fog of gnats you've grown accustomed to still affects you. For what it's worth, the ads aren't for you. They're for anybody or anything that might click them, knowingly or not, and that's not you.

deepserket 3y ago

Isn't boosting websites without ads against their interest?

wood_spirit 3y ago
This is what I was hoping all the replies would be about :) Is monetising the web through advertising the reason these scraper sites exist? And removing that would make scraping sites an expense with costs rather than profits? Perhaps google could do this. Ads on third party sites are probably a small part of their income compared to sponsored ads in search results, so they could be sacrificed and they’d take down their competitors who rely more on the website ads because they don’t have the search? However, they have no need to compete on search quality at this time. If another player in the search and browser wars started downweighimg ad heavy sites then this might make google have to reluctantly follow suit. But you can imagine the FUD campaigns to paint such an actor as killing the internet and being the bad guy etc…