4 ms·

> “Edit: maybe you are thinking about the AI deal which is exclusive to Google. That's not the same thing as search engine indexing.” @mastazi that’s what I’m

> “Edit: maybe you are thinking about the AI deal which is exclusive to Google. That's not the same thing as search engine indexing.” @mastazi that’s what I’m talking about, and I think your AI vs. indexing nuance is incorrect. I wasn’t sure, so I just did a quick N=1 verification: searching for the name of a random 1week-old popular Reddit post with a precise unique title, - Insta-found it on Goog as top result - Didn't find it on DDG, with or without site:reddit.com Looks like sibling comment from @cpressland (thanks!) is correct: as of today and until other search engines sign licensing agreements with Reddit, “non-Google search engines cannot get new results from Reddit”. See https://www.reddit.com/robots.txt https://www.reddit.com/robots.txt , which links to https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy https://support.reddithelp.com/hc/en-us/articles/26410290525... , section “Reddit may license public content for commercial or non-commercial use”

esperent 2y ago
I thought that DDG and Kagi get a large amount of their results from Google? Do these have reddit results stripped out? > a random 1week-old popular Reddit post with a precise unique title While a week is a long time, is it possible that the other search engines just hadn't got around to indexing it yet?
- ronjouch 2y ago
  DDG gets a large amount of their results from Bing: https://duckduckgo.com/duckduckgo-help-pages/results/sources/ https://duckduckgo.com/duckduckgo-help-pages/results/sources... Kagi I don’t know much about, but quickly reading about it, seems to indeed pay Goog. > “While a week is a long time, is it possible that the other search engines just hadn't got around to indexing it yet?” I doubt it, search engines these days are much faster, and this is double-confirmed by 1. articles linked in sibling comments, 2. reading https://www.reddit.com/robots.txt https://www.reddit.com/robots.txt
mastazi 2y ago
you are right, and I was wrong; I found this article by Ars Technica that highlights the consequences of the deal for search engines: https://arstechnica.com/gadgets/2024/07/non-google-search-engines-blocked-from-showing-recent-reddit-results/ https://arstechnica.com/gadgets/2024/07/non-google-search-en... so Reddit, the website co created by Aaron Swartz, is now a walled garden. Wow.

esperent 2y ago

I thought that DDG and Kagi get a large amount of their results from Google? Do these have reddit results stripped out? > a random 1week-old popular Reddit post with a precise unique title While a week is a long time, is it possible that the other search engines just hadn't got around to indexing it yet?

ronjouch 2y ago
DDG gets a large amount of their results from Bing: https://duckduckgo.com/duckduckgo-help-pages/results/sources/ https://duckduckgo.com/duckduckgo-help-pages/results/sources... Kagi I don’t know much about, but quickly reading about it, seems to indeed pay Goog. > “While a week is a long time, is it possible that the other search engines just hadn't got around to indexing it yet?” I doubt it, search engines these days are much faster, and this is double-confirmed by 1. articles linked in sibling comments, 2. reading https://www.reddit.com/robots.txt https://www.reddit.com/robots.txt

mastazi 2y ago

you are right, and I was wrong; I found this article by Ars Technica that highlights the consequences of the deal for search engines: https://arstechnica.com/gadgets/2024/07/non-google-search-engines-blocked-from-showing-recent-reddit-results/ https://arstechnica.com/gadgets/2024/07/non-google-search-en... so Reddit, the website co created by Aaron Swartz, is now a walled garden. Wow.