13 ms·

This author is in his own little bubble and doesn't understand the vast amount of blog-repost spam that google has to deal with. The way their algorithm most li

This author is in his own little bubble and doesn't understand the vast amount of blog-repost spam that google has to deal with. The way their algorithm most likely deals with this is a mixture of domain rank + tenure... how long has this copy of this article existed on this domain, and can we be sure this is the original copy? The author says the article was removed in 2006 ("[...] posts, were not accessible anymore") and then he re-posted the article at a new domain in 2013. That means any copy/crawl/repost of the article from 2006-2012 is now the oldest living, and thus "original", version of the article. His 2013 repost was seen as just another blog-spam copy. Google is not forgetting the old web unless we see evidence of content disappearing from the index that have been consistently hosted at the same domain & URL since their original posts. Unless you properly 301 your URLs to new locations and consistently host your content, it's a guessing game for the crawler to determine where the original content has moved to.

Majestic121 8y ago
The first paragraph of the article mention the story of Tim Bray[0], which is exactly about this : Google forgetting an article which did not change location. [0] : https://www.tbray.org/ongoing/When/201x/2018/01/15/Google-is-losing-its-memory https://www.tbray.org/ongoing/When/201x/2018/01/15/Google-is...
EastSmith 8y ago
Yeah, but how DDG was able to show "the new original"?
- giancarlostoro 8y ago
  DDG isn't only using Google, it uses other search engines too.
  - mda 8y ago
    Mostly Bing.
- fencepost 8y ago
  DDG's primary search is actually Bing IIRC.
kickscondor 8y ago
Part of the problem is that their algorithm has become weighted against blogs and personal websites. > Rumors spread that large link pages (for surfing) might be considered “link farms” (and yes on SEO sites they were but these things eventually trickle down to little personal site webmasters too) so these started to be phased out. Then the worry was Blogrolls might be considered link farms so they slowly started to be phased out. Then the biggie: when Google deliberately filtered out all the free hosted sites from the SERP’s (they were not removed completely just sent back to page 10 or so of the Google SERP’s) and traffic to Tripod and Geocities plummeted. Why? Because they were taking up space in the first 20 organic returns knocking out corporate and commercial sites and the sites likely to become paying customers were complaining. https://ramblinggit.com/2018/08/when-the-social-silos-fall/ https://ramblinggit.com/2018/08/when-the-social-silos-fall/ SEO seems to have become a huge obstacle course that smaller websites can't play.
- benologist 8y ago
  That's exactly what it is, and Google's also incentivizing many low-quality sites to engage disproportionately in SEO to boost their Google Adsense earnings too.
- LeifCarrotson 8y ago
  You're jumping from describing observable results to a state of mind or motive which you can't observe. > Then the worry was Blogrolls might be considered link farms so they slowly started to be phased out. Then the biggie: when Google deliberately filtered out all the free hosted sites from the SERP’s... That's all observable fact. Why? Because they were taking up space in the first 20 organic returns knocking out corporate and commercial sites and the sites likely to become paying customers were complaining. I think the more reasonable, less diabolical motive was that the blogs and free hosted sites were largely link farms that no one wanted to visit. It sucks for the few legitimate pages on those platforms, but when most of the legitimate page is the rare gem in a minefield of automated copies of other blogs, just with SEO links and ads inserted. It's like a comments section: without moderation or captchas or both, a "thriving local community" on, say, a small town news site can be overwhelmed by automated pharmaceuticals spam. Then the newspaper kills the comment section, not out of any malice towards the original community but because they don't want to deal with the spam. And yeah, dealing with spam and black hat SEO does take resources. If you (or worse, your chosen blog host) don't keep the weeds down, soon your pasture will be overrun and burned off.
xorand 8y ago
Yesterday I noticed that Google Scholar forgot one of my articles from 2018, on arXiv. See: https://scholar.google.com/scholar?q=arXiv%3A1811.04960 https://scholar.google.com/scholar?q=arXiv%3A1811.04960 Google Scholar is not the same as Google Search, which can still find it https://www.google.com/search?q=arXiv%3A1811.04960 https://www.google.com/search?q=arXiv%3A1811.04960 For how long, I have no idea. The article was at the same link all the time and arXiv is very reputable.
- eruci 8y ago
  I also noticed that all our scholarly articles are gone from Google Scholar. The only thing there is our two highly cited books. https://scholar.google.ca/scholar?hl=en&as_sdt=0%2C5&q=site%3Arepo.risat.org&oq=repo https://scholar.google.ca/scholar?hl=en&as_sdt=0%2C5&q=site%... We've come to rely on Google too much, so much that if you are not on Google you don't exist. That's a problem with researchers that are looking for articles to cite.
  - xorand 8y ago
    Is this recent? In my case I noticed it yesterday.
    - eruci 8y ago
      I also just noticed it. No idea when the rest of the papers were dropped.
      - xorand 8y ago
        I sent today a message to Google Scholar with this https://support.google.com/scholar/contact/general https://support.google.com/scholar/contact/general
    - eruci 8y ago
      The articles are still on Google Search though: https://www.google.com/search?q=site%3Arepo.risat.org https://www.google.com/search?q=site%3Arepo.risat.org Fingers crossed they don't get dropped from the main index too.
SquareWheel 8y ago
They even have a tool just for this: canonical URLs. It lets websites specify which version is the source/canonical version and avoids the old copy indexing.
deleted 8y ago
[deleted]
api 8y ago
So as with all other open systems, spam is destroying the web.
- zrm 8y ago
  It seems like the opposite actually -- spam is destroying Google. They're so big that it's worth blackhats spending significant resources to game their algorithm. That induced them to implement a spam filter which is now discarding the ham along with the spam. Which means that smaller search engines that aren't being targeted by spammers are now giving better results. That is a major long-term problem for Google if they can't avoid throwing the baby out with the bathwater like this. People only use Google because it has historically had the best results. They'll get some way on inertia now, but that doesn't last forever. They need to fix this or they're ultimately in trouble, and we could be heading for a landscape where being a search engine above a threshold size is a liability.
  - revvx 8y ago
    IME, it's not blackhats anymore causing the problem. It's (legitimate, but shady) marketing agencies and startups handling thousands of customers and with deep pockets to do SEO research.
    - JohnFen 8y ago
      I count those agencies as a variety of blackhat.
      - rolph 8y ago
        blackhat is as a blackhat does, its no difference how or why you screw users in someway by failing to be forthright and of candor. if you do it constituatively thats blackhat.
- andrewprock 8y ago
  The web is fine, and search is fine. It's specifically Google search that's being destroyed by spam. It's odd to put forward the hypothesis that DuckDuckGo is now better at search (aggregation) than Google is at search. But that seems to be where we have landed.
founderling 8y ago
Here is an example: http://www.gnoosic.com/discussion/metallica__5.html http://www.gnoosic.com/discussion/metallica__5.html No matter how you search for the content on Google, nothing comes up: https://www.google.com/search?q="Metallica+only+played+2+songs+from+St.+Anger" https://www.google.com/search?q="Metallica+only+played+2+son... DuckDuckGo has it: https://duckduckgo.com/?q="Metallica+only+played+2+songs+from+St.+Anger" https://duckduckgo.com/?q="Metallica+only+played+2+songs+fro... I checked the wayback machine and the content has constantly been on that url for over 10 years. This is the first example of an old forum page I tried after reading the article. So I tend to think it's true. Google is discarding the "classic" web.
- vissi 8y ago
  Well, search may have bugs (or undocumented features) too, I have googled content from other pages on this site (related to Metallica). Page 6 for example: https://www.google.com/search?q=%22Listen+up+you+fags+metallica+IS+THE+MOST+OVERATED%22&oq=%22Listen+up+you+fags+metallica+IS+THE+MOST+OVERATED%22&aqs=chrome..69i57.881j0j7&sourceid=chrome&ie=UTF-8 https://www.google.com/search?q=%22Listen+up+you+fags+metall...
  - founderling 8y ago
    I don't think the articles premise is that Google axed all content older then 5 years or so. But that it gradually discards old unique content. Which goes against the original mission of Google to "organize the world's information and make it universally accessible". A "bug" could be an option, but I don't expect that to be the reason. It's too easy to find examples of forgotten content. And I don't think a bug of that magnitude in Googles core business would go unnoticed.
    - misterman0 8y ago
      >> Googles core business Which core business are you referring to?
      - 8y ago
rsync 8y ago
"Google is not forgetting the old web unless we see evidence of content disappearing from the index that have been consistently hosted at the same domain & URL since their original posts." I can, very loosely and anecdotally, confirm. My personal website has been online for about 20 years and I just picked some deep strings of text and searched for them and google has the whole thing indexed just fine ...
- deleted 8y ago
  [deleted]
gambler 8y ago
>The way their algorithm most likely deals with this is a mixture of domain rank + tenure... how long has this copy of this article existed on this domain, and can we be sure this is the original copy? This rationalization doesn't change the fact that it's incresingly hard or impossible to find certain things on Google, that they are effectively biased against certain types of websites and certain types of pages (even when the content is perfectly good) and that other search engines seem to be able to deal with these issues much better.
ashelmire 8y ago
Google has weird rules and inconsistent indexing. I recently published a two-part article on using graphql/apollo with react and rails; it never indexed the first part (the graphql / rails bit) but did index the second part (react and apollo). And in fact, searching for with "graphql rails react apollo" still doesn't show any results for this page on google despite ostensibly being indexed, but it shows up on duckduckgo. And looking over the results on google, only ~4 are actually relevant to the topic, so it's not like good content is being shown.
Wowfunhappy 8y ago
If the originally indexed copy no longer exists, Google shouldn't down-rank a reposted version!
jeena 8y ago
I tested it with an article on my own website from 2003. I first posted it on http://jeenaparadies.de/artikel/webdesign http://jeenaparadies.de/artikel/webdesign then I had a 301 redirect there for a couple of years to http://jeenaparadies.net/artikel/webdesign http://jeenaparadies.net/artikel/webdesign until I stopped paying for the .de domain. About 5 years ago I made another 301 redirect to http://paradies.jeena.net/artikel/webdesign http://paradies.jeena.net/artikel/webdesign which is still in place. DDG finds it but not Google and actually neither does Bing.

Majestic121 8y ago

The first paragraph of the article mention the story of Tim Bray[0], which is exactly about this : Google forgetting an article which did not change location. [0] : https://www.tbray.org/ongoing/When/201x/2018/01/15/Google-is-losing-its-memory https://www.tbray.org/ongoing/When/201x/2018/01/15/Google-is...

EastSmith 8y ago

Yeah, but how DDG was able to show "the new original"?

giancarlostoro 8y ago
DDG isn't only using Google, it uses other search engines too.
- mda 8y ago
  Mostly Bing.
fencepost 8y ago
DDG's primary search is actually Bing IIRC.

kickscondor 8y ago

Part of the problem is that their algorithm has become weighted against blogs and personal websites. > Rumors spread that large link pages (for surfing) might be considered “link farms” (and yes on SEO sites they were but these things eventually trickle down to little personal site webmasters too) so these started to be phased out. Then the worry was Blogrolls might be considered link farms so they slowly started to be phased out. Then the biggie: when Google deliberately filtered out all the free hosted sites from the SERP’s (they were not removed completely just sent back to page 10 or so of the Google SERP’s) and traffic to Tripod and Geocities plummeted. Why? Because they were taking up space in the first 20 organic returns knocking out corporate and commercial sites and the sites likely to become paying customers were complaining. https://ramblinggit.com/2018/08/when-the-social-silos-fall/ https://ramblinggit.com/2018/08/when-the-social-silos-fall/ SEO seems to have become a huge obstacle course that smaller websites can't play.

benologist 8y ago
That's exactly what it is, and Google's also incentivizing many low-quality sites to engage disproportionately in SEO to boost their Google Adsense earnings too.
LeifCarrotson 8y ago
You're jumping from describing observable results to a state of mind or motive which you can't observe. > Then the worry was Blogrolls might be considered link farms so they slowly started to be phased out. Then the biggie: when Google deliberately filtered out all the free hosted sites from the SERP’s... That's all observable fact. Why? Because they were taking up space in the first 20 organic returns knocking out corporate and commercial sites and the sites likely to become paying customers were complaining. I think the more reasonable, less diabolical motive was that the blogs and free hosted sites were largely link farms that no one wanted to visit. It sucks for the few legitimate pages on those platforms, but when most of the legitimate page is the rare gem in a minefield of automated copies of other blogs, just with SEO links and ads inserted. It's like a comments section: without moderation or captchas or both, a "thriving local community" on, say, a small town news site can be overwhelmed by automated pharmaceuticals spam. Then the newspaper kills the comment section, not out of any malice towards the original community but because they don't want to deal with the spam. And yeah, dealing with spam and black hat SEO does take resources. If you (or worse, your chosen blog host) don't keep the weeds down, soon your pasture will be overrun and burned off.

xorand 8y ago

Yesterday I noticed that Google Scholar forgot one of my articles from 2018, on arXiv. See: https://scholar.google.com/scholar?q=arXiv%3A1811.04960 https://scholar.google.com/scholar?q=arXiv%3A1811.04960 Google Scholar is not the same as Google Search, which can still find it https://www.google.com/search?q=arXiv%3A1811.04960 https://www.google.com/search?q=arXiv%3A1811.04960 For how long, I have no idea. The article was at the same link all the time and arXiv is very reputable.

eruci 8y ago
I also noticed that all our scholarly articles are gone from Google Scholar. The only thing there is our two highly cited books. https://scholar.google.ca/scholar?hl=en&as_sdt=0%2C5&q=site%3Arepo.risat.org&oq=repo https://scholar.google.ca/scholar?hl=en&as_sdt=0%2C5&q=site%... We've come to rely on Google too much, so much that if you are not on Google you don't exist. That's a problem with researchers that are looking for articles to cite.
- xorand 8y ago
  Is this recent? In my case I noticed it yesterday.
  - eruci 8y ago
    I also just noticed it. No idea when the rest of the papers were dropped.
    - xorand 8y ago
      I sent today a message to Google Scholar with this https://support.google.com/scholar/contact/general https://support.google.com/scholar/contact/general
  - eruci 8y ago
    The articles are still on Google Search though: https://www.google.com/search?q=site%3Arepo.risat.org https://www.google.com/search?q=site%3Arepo.risat.org Fingers crossed they don't get dropped from the main index too.

SquareWheel 8y ago

They even have a tool just for this: canonical URLs. It lets websites specify which version is the source/canonical version and avoids the old copy indexing.

deleted 8y ago

[deleted]

api 8y ago

So as with all other open systems, spam is destroying the web.

zrm 8y ago
It seems like the opposite actually -- spam is destroying Google. They're so big that it's worth blackhats spending significant resources to game their algorithm. That induced them to implement a spam filter which is now discarding the ham along with the spam. Which means that smaller search engines that aren't being targeted by spammers are now giving better results. That is a major long-term problem for Google if they can't avoid throwing the baby out with the bathwater like this. People only use Google because it has historically had the best results. They'll get some way on inertia now, but that doesn't last forever. They need to fix this or they're ultimately in trouble, and we could be heading for a landscape where being a search engine above a threshold size is a liability.
- revvx 8y ago
  IME, it's not blackhats anymore causing the problem. It's (legitimate, but shady) marketing agencies and startups handling thousands of customers and with deep pockets to do SEO research.
  - JohnFen 8y ago
    I count those agencies as a variety of blackhat.
    - rolph 8y ago
      blackhat is as a blackhat does, its no difference how or why you screw users in someway by failing to be forthright and of candor. if you do it constituatively thats blackhat.
andrewprock 8y ago
The web is fine, and search is fine. It's specifically Google search that's being destroyed by spam. It's odd to put forward the hypothesis that DuckDuckGo is now better at search (aggregation) than Google is at search. But that seems to be where we have landed.

founderling 8y ago

Here is an example: http://www.gnoosic.com/discussion/metallica__5.html http://www.gnoosic.com/discussion/metallica__5.html No matter how you search for the content on Google, nothing comes up: https://www.google.com/search?q="Metallica+only+played+2+songs+from+St.+Anger" https://www.google.com/search?q="Metallica+only+played+2+son... DuckDuckGo has it: https://duckduckgo.com/?q="Metallica+only+played+2+songs+from+St.+Anger" https://duckduckgo.com/?q="Metallica+only+played+2+songs+fro... I checked the wayback machine and the content has constantly been on that url for over 10 years. This is the first example of an old forum page I tried after reading the article. So I tend to think it's true. Google is discarding the "classic" web.

vissi 8y ago
Well, search may have bugs (or undocumented features) too, I have googled content from other pages on this site (related to Metallica). Page 6 for example: https://www.google.com/search?q=%22Listen+up+you+fags+metallica+IS+THE+MOST+OVERATED%22&oq=%22Listen+up+you+fags+metallica+IS+THE+MOST+OVERATED%22&aqs=chrome..69i57.881j0j7&sourceid=chrome&ie=UTF-8 https://www.google.com/search?q=%22Listen+up+you+fags+metall...
- founderling 8y ago
  I don't think the articles premise is that Google axed all content older then 5 years or so. But that it gradually discards old unique content. Which goes against the original mission of Google to "organize the world's information and make it universally accessible". A "bug" could be an option, but I don't expect that to be the reason. It's too easy to find examples of forgotten content. And I don't think a bug of that magnitude in Googles core business would go unnoticed.
  - misterman0 8y ago
    >> Googles core business Which core business are you referring to?
    - 8y ago

rsync 8y ago

"Google is not forgetting the old web unless we see evidence of content disappearing from the index that have been consistently hosted at the same domain & URL since their original posts." I can, very loosely and anecdotally, confirm. My personal website has been online for about 20 years and I just picked some deep strings of text and searched for them and google has the whole thing indexed just fine ...

deleted 8y ago
[deleted]

gambler 8y ago

>The way their algorithm most likely deals with this is a mixture of domain rank + tenure... how long has this copy of this article existed on this domain, and can we be sure this is the original copy? This rationalization doesn't change the fact that it's incresingly hard or impossible to find certain things on Google, that they are effectively biased against certain types of websites and certain types of pages (even when the content is perfectly good) and that other search engines seem to be able to deal with these issues much better.

ashelmire 8y ago

Google has weird rules and inconsistent indexing. I recently published a two-part article on using graphql/apollo with react and rails; it never indexed the first part (the graphql / rails bit) but did index the second part (react and apollo). And in fact, searching for with "graphql rails react apollo" still doesn't show any results for this page on google despite ostensibly being indexed, but it shows up on duckduckgo. And looking over the results on google, only ~4 are actually relevant to the topic, so it's not like good content is being shown.

Wowfunhappy 8y ago

If the originally indexed copy no longer exists, Google shouldn't down-rank a reposted version!

jeena 8y ago

I tested it with an article on my own website from 2003. I first posted it on http://jeenaparadies.de/artikel/webdesign http://jeenaparadies.de/artikel/webdesign then I had a 301 redirect there for a couple of years to http://jeenaparadies.net/artikel/webdesign http://jeenaparadies.net/artikel/webdesign until I stopped paying for the .de domain. About 5 years ago I made another 301 redirect to http://paradies.jeena.net/artikel/webdesign http://paradies.jeena.net/artikel/webdesign which is still in place. DDG finds it but not Google and actually neither does Bing.