4 ms·

Every search engine other than Google has stopped indexing pages from Reddit. Google has not commented on whether they plan to respect it. Rich Results[0] say

Every search engine other than Google has stopped indexing pages from Reddit. Google has not commented on whether they plan to respect it. Rich Results[0] say they're using a version from June 25. The new version was last modified July 1. [0]: https://search.google.com/test/rich-results https://search.google.com/test/rich-results

fragmede 2y ago
Google is paying Reddit $60 million to not have to respect it. The media reports it's for training data for AI, but none of us have read that contract.
- skilled 2y ago
  Yes, it makes sense from a 600 million monthly visitors point of view, but that would still be a pretty interesting development if they don't respect it.
  - fragmede 2y ago
    if they're getting the data directly from Reddit then they can respect it while still having the content because they're not crawling it via http.
- darreninthenet 2y ago
  Isn't this monopoly abuse?
  - fragmede 2y ago
    any other company, specifically Microsoft/Bing and OpenAI, are free to pay reddit for the same privilege, so I don't see how it could be considered that.
    - darreninthenet 2y ago
      I actually meant from Reddit as they hold a monopoly on online forums imo, but I'm no expert in this, was just wondering
altdataseller 2y ago
Bing is still indexing it. As is DDG. https://www.bing.com/search?q=reddit+travel&qs=n&form=QBRE&sp=-1&ghc=1&lq=0&pq=reddit+travel&sc=9-13&sk=&cvid=C9E48332A62E44E29BCA9E364B91B635&ghsh=0&ghacc=0&ghpl= https://www.bing.com/search?q=reddit+travel&qs=n&form=QBRE&s...
- skilled 2y ago
  They are not indexing new pages. I should have made that more clear.
altdataseller 2y ago
They can and will probably respect it and still be able to have reddit results. “User-agent: * Disallow: /“ in the `robots.txt` simply instructs all web crawlers not to crawl any pages on the website, not whether to show pages in search results. Of course new pages wont be shown if they cant be crawled if theres no other way you can get it.. But that line simply prevents crawlers that abide by robots.txt from retrieving the content of any pages. But Google doesnt need to crawl Reddit anymore. Reddit is directly giving the data to them to serve to users! Bing, DDG and any other search engines now are basically forced to start paying Reddit. And presumably, the Reddit execs have calculated that future revenue will be more because of this. Don’t be fooled: This isnt a “good for internet, morality” decision. Reddit is a public company now and has shareholders. They are doing this because it will equate to more $$$$

fragmede 2y ago

Google is paying Reddit $60 million to not have to respect it. The media reports it's for training data for AI, but none of us have read that contract.

skilled 2y ago
Yes, it makes sense from a 600 million monthly visitors point of view, but that would still be a pretty interesting development if they don't respect it.
- fragmede 2y ago
  if they're getting the data directly from Reddit then they can respect it while still having the content because they're not crawling it via http.
darreninthenet 2y ago
Isn't this monopoly abuse?
- fragmede 2y ago
  any other company, specifically Microsoft/Bing and OpenAI, are free to pay reddit for the same privilege, so I don't see how it could be considered that.
  - darreninthenet 2y ago
    I actually meant from Reddit as they hold a monopoly on online forums imo, but I'm no expert in this, was just wondering

altdataseller 2y ago

Bing is still indexing it. As is DDG. https://www.bing.com/search?q=reddit+travel&qs=n&form=QBRE&sp=-1&ghc=1&lq=0&pq=reddit+travel&sc=9-13&sk=&cvid=C9E48332A62E44E29BCA9E364B91B635&ghsh=0&ghacc=0&ghpl= https://www.bing.com/search?q=reddit+travel&qs=n&form=QBRE&s...

skilled 2y ago
They are not indexing new pages. I should have made that more clear.

altdataseller 2y ago

They can and will probably respect it and still be able to have reddit results. “User-agent: * Disallow: /“ in the `robots.txt` simply instructs all web crawlers not to crawl any pages on the website, not whether to show pages in search results. Of course new pages wont be shown if they cant be crawled if theres no other way you can get it.. But that line simply prevents crawlers that abide by robots.txt from retrieving the content of any pages. But Google doesnt need to crawl Reddit anymore. Reddit is directly giving the data to them to serve to users! Bing, DDG and any other search engines now are basically forced to start paying Reddit. And presumably, the Reddit execs have calculated that future revenue will be more because of this. Don’t be fooled: This isnt a “good for internet, morality” decision. Reddit is a public company now and has shareholders. They are doing this because it will equate to more $$$$