3 ms·

Google is currently the only search engine allowed to crawl Reddit, which sometimes yields good original user content of actual { non-blogspam, non-SEOed-to-hel

Google is currently the only search engine allowed to crawl Reddit, which sometimes yields good original user content of actual { non-blogspam, non-SEOed-to-hell, non-AI } value.

deleted 2y ago
[deleted]
mastazi 2y ago
All search engines include Reddit results by default and you can usually refine by adding some param like site:reddit.com which works the same in Google as in other search engines e.g. https://duckduckgo.com/?t=ffab&q=filter+coffee+site%3Areddit.com&ia=web https://duckduckgo.com/?t=ffab&q=filter+coffee+site%3Areddit... Edit: maybe you are thinking about the AI deal which is exclusive to Google. That's not the same thing as search engine indexing https://www.cbsnews.com/news/google-reddit-60-million-deal-ai-training/ https://www.cbsnews.com/news/google-reddit-60-million-deal-a...
- deleted 2y ago
  [deleted]
- cpressland 2y ago
  While I couldn’t find a better document: https://www.tomsguide.com/computing/search-engines/google-is-now-the-only-search-engine-that-shows-recent-reddit-posts-in-results-heres-why https://www.tomsguide.com/computing/search-engines/google-is... describes how non-Google search engines cannot get new results from Reddit.
  - mastazi 2y ago
    Yes, I was wrong. I found this one as well https://arstechnica.com/gadgets/2024/07/non-google-search-engines-blocked-from-showing-recent-reddit-results/ https://arstechnica.com/gadgets/2024/07/non-google-search-en...
- ronjouch 2y ago
  > “Edit: maybe you are thinking about the AI deal which is exclusive to Google. That's not the same thing as search engine indexing.” @mastazi that’s what I’m talking about, and I think your AI vs. indexing nuance is incorrect. I wasn’t sure, so I just did a quick N=1 verification: searching for the name of a random 1week-old popular Reddit post with a precise unique title, - Insta-found it on Goog as top result - Didn't find it on DDG, with or without site:reddit.com Looks like sibling comment from @cpressland (thanks!) is correct: as of today and until other search engines sign licensing agreements with Reddit, “non-Google search engines cannot get new results from Reddit”. See https://www.reddit.com/robots.txt https://www.reddit.com/robots.txt , which links to https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy https://support.reddithelp.com/hc/en-us/articles/26410290525... , section “Reddit may license public content for commercial or non-commercial use”
  - esperent 2y ago
    I thought that DDG and Kagi get a large amount of their results from Google? Do these have reddit results stripped out? > a random 1week-old popular Reddit post with a precise unique title While a week is a long time, is it possible that the other search engines just hadn't got around to indexing it yet?
    - ronjouch 2y ago
      DDG gets a large amount of their results from Bing: https://duckduckgo.com/duckduckgo-help-pages/results/sources/ https://duckduckgo.com/duckduckgo-help-pages/results/sources... Kagi I don’t know much about, but quickly reading about it, seems to indeed pay Goog. > “While a week is a long time, is it possible that the other search engines just hadn't got around to indexing it yet?” I doubt it, search engines these days are much faster, and this is double-confirmed by 1. articles linked in sibling comments, 2. reading https://www.reddit.com/robots.txt https://www.reddit.com/robots.txt
  - mastazi 2y ago
    you are right, and I was wrong; I found this article by Ars Technica that highlights the consequences of the deal for search engines: https://arstechnica.com/gadgets/2024/07/non-google-search-engines-blocked-from-showing-recent-reddit-results/ https://arstechnica.com/gadgets/2024/07/non-google-search-en... so Reddit, the website co created by Aaron Swartz, is now a walled garden. Wow.
- cma 2y ago
  ChatGPT search isn't allowed to index reddit
taikahessu 2y ago
Funny, that is indeed true, didn't know that.
qwertox 2y ago
This partnership has created a problem, though: My Google is set to German. Apparently Reddit has autotranslated all their content into German. If I do a Google search for "ssd zfs pool" the 4th result is "SSD-Pool eine schlechte Idee? : r/Proxmox" and this links to `https://www.reddit.com/r/Proxmox/comments/12a5abh/ssd_pool_a_bad_idea/?tl=de https://www.reddit.com/r/Proxmox/comments/12a5abh/ssd_pool_a...`. The Reddit page is in German, but as you may have noticed, the URL has `?tl=de` appended, while it contains `ssd_pool_a_bad_idea` in the path. If I remove the `?tl=de`, I get the original version, in English. This means that what Google crawled, what it has in its index, was already in German. So Reddit translated the original page into German, then made it accessible for Google to index it. For me this causes the problem that I am now getting a lot of AI-translated Reddit content, even though I'd really like to have the English version to begin with, because I assume that it won't contain translation errors. I mean, the translation is very good, you probably wouldn't notice that it is one, but still... https://www.reddit.com/r/Proxmox/comments/12a5abh/ssd_pool_a_bad_idea/ https://www.reddit.com/r/Proxmox/comments/12a5abh/ssd_pool_a... https://www.reddit.com/r/Proxmox/comments/12a5abh/ssd_pool_a_bad_idea/?tl=de https://www.reddit.com/r/Proxmox/comments/12a5abh/ssd_pool_a...
- ReadCarlBarks 2y ago
  Add this to your uBlock Origin filters: reddit.com$removeparam=tl
inetknght 2y ago
> Google is currently the only search engine allowed to crawl Reddit Stop using Reddit. Reddit is already following the path toward nothing but SEO spam and malware. Use Hacker News or Fediverse.

Google is currently the only search engine allowed to crawl Reddit, which sometimes yields good original user content of actual { non-blogspam, non-SEOed-to-hell, non-AI } value.

deleted 2y ago

[deleted]

mastazi 2y ago

All search engines include Reddit results by default and you can usually refine by adding some param like site:reddit.com which works the same in Google as in other search engines e.g. https://duckduckgo.com/?t=ffab&q=filter+coffee+site%3Areddit.com&ia=web https://duckduckgo.com/?t=ffab&q=filter+coffee+site%3Areddit... Edit: maybe you are thinking about the AI deal which is exclusive to Google. That's not the same thing as search engine indexing https://www.cbsnews.com/news/google-reddit-60-million-deal-ai-training/ https://www.cbsnews.com/news/google-reddit-60-million-deal-a...

deleted 2y ago
[deleted]
cpressland 2y ago
While I couldn’t find a better document: https://www.tomsguide.com/computing/search-engines/google-is-now-the-only-search-engine-that-shows-recent-reddit-posts-in-results-heres-why https://www.tomsguide.com/computing/search-engines/google-is... describes how non-Google search engines cannot get new results from Reddit.
- mastazi 2y ago
  Yes, I was wrong. I found this one as well https://arstechnica.com/gadgets/2024/07/non-google-search-engines-blocked-from-showing-recent-reddit-results/ https://arstechnica.com/gadgets/2024/07/non-google-search-en...
ronjouch 2y ago
> “Edit: maybe you are thinking about the AI deal which is exclusive to Google. That's not the same thing as search engine indexing.” @mastazi that’s what I’m talking about, and I think your AI vs. indexing nuance is incorrect. I wasn’t sure, so I just did a quick N=1 verification: searching for the name of a random 1week-old popular Reddit post with a precise unique title, - Insta-found it on Goog as top result - Didn't find it on DDG, with or without site:reddit.com Looks like sibling comment from @cpressland (thanks!) is correct: as of today and until other search engines sign licensing agreements with Reddit, “non-Google search engines cannot get new results from Reddit”. See https://www.reddit.com/robots.txt https://www.reddit.com/robots.txt , which links to https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy https://support.reddithelp.com/hc/en-us/articles/26410290525... , section “Reddit may license public content for commercial or non-commercial use”
- esperent 2y ago
  I thought that DDG and Kagi get a large amount of their results from Google? Do these have reddit results stripped out? > a random 1week-old popular Reddit post with a precise unique title While a week is a long time, is it possible that the other search engines just hadn't got around to indexing it yet?
  - ronjouch 2y ago
    DDG gets a large amount of their results from Bing: https://duckduckgo.com/duckduckgo-help-pages/results/sources/ https://duckduckgo.com/duckduckgo-help-pages/results/sources... Kagi I don’t know much about, but quickly reading about it, seems to indeed pay Goog. > “While a week is a long time, is it possible that the other search engines just hadn't got around to indexing it yet?” I doubt it, search engines these days are much faster, and this is double-confirmed by 1. articles linked in sibling comments, 2. reading https://www.reddit.com/robots.txt https://www.reddit.com/robots.txt
- mastazi 2y ago
  you are right, and I was wrong; I found this article by Ars Technica that highlights the consequences of the deal for search engines: https://arstechnica.com/gadgets/2024/07/non-google-search-engines-blocked-from-showing-recent-reddit-results/ https://arstechnica.com/gadgets/2024/07/non-google-search-en... so Reddit, the website co created by Aaron Swartz, is now a walled garden. Wow.
cma 2y ago
ChatGPT search isn't allowed to index reddit

taikahessu 2y ago

Funny, that is indeed true, didn't know that.

qwertox 2y ago

This partnership has created a problem, though: My Google is set to German. Apparently Reddit has autotranslated all their content into German. If I do a Google search for "ssd zfs pool" the 4th result is "SSD-Pool eine schlechte Idee? : r/Proxmox" and this links to `https://www.reddit.com/r/Proxmox/comments/12a5abh/ssd_pool_a_bad_idea/?tl=de https://www.reddit.com/r/Proxmox/comments/12a5abh/ssd_pool_a...`. The Reddit page is in German, but as you may have noticed, the URL has `?tl=de` appended, while it contains `ssd_pool_a_bad_idea` in the path. If I remove the `?tl=de`, I get the original version, in English. This means that what Google crawled, what it has in its index, was already in German. So Reddit translated the original page into German, then made it accessible for Google to index it. For me this causes the problem that I am now getting a lot of AI-translated Reddit content, even though I'd really like to have the English version to begin with, because I assume that it won't contain translation errors. I mean, the translation is very good, you probably wouldn't notice that it is one, but still... https://www.reddit.com/r/Proxmox/comments/12a5abh/ssd_pool_a_bad_idea/ https://www.reddit.com/r/Proxmox/comments/12a5abh/ssd_pool_a... https://www.reddit.com/r/Proxmox/comments/12a5abh/ssd_pool_a_bad_idea/?tl=de https://www.reddit.com/r/Proxmox/comments/12a5abh/ssd_pool_a...

ReadCarlBarks 2y ago
Add this to your uBlock Origin filters: reddit.com$removeparam=tl

inetknght 2y ago

> Google is currently the only search engine allowed to crawl Reddit Stop using Reddit. Reddit is already following the path toward nothing but SEO spam and malware. Use Hacker News or Fediverse.