3 ms·
It's also not much use to anyone who doesn't use Google ever since Reddit started blocking all crawlers besides Googlebot. Old cached results might still show u
by jsheard 2y ago
It's also not much use to anyone who doesn't use Google ever since Reddit started blocking all crawlers besides Googlebot. Old cached results might still show up in Bing/DDG/Kagi but they can't index any of the newer stuff.
- alberto_balsam 2y agoWhat makes you say this? I just tried a few Reddit searches about events from the last few days and they work as expected (tested in Kagi)
- jsheard 2y agohttps://www.404media.co/google-is-the-only-search-engine-that-works-on-reddit-now-thanks-to-ai-deal/ https://www.404media.co/google-is-the-only-search-engine-tha... https://www.reddit.com/robots.txt https://www.reddit.com/robots.txt Reddit serves a different robots.txt to Googlebot, you can see a snippet of it in Googles summary. If Kagi is getting recent Reddit results then they must be either ignoring robots.txt, using Google as a backend, or also paying Reddit for access like Google. Bing and DuckDuckGo are certainly still locked out, I just tried searching for "reddit hurricane milton" on both and none of the results are actually from Reddit.
- SavioMak 2y agoKagi didn't make it explicit, but based on what they wrote I think they API call Google and other search engines to show as part of their results. https://help.kagi.com/kagi/search-details/search-sources.html https://help.kagi.com/kagi/search-details/search-sources.htm...