3 ms·
"to tell crawler to not crawl" which can be ignored AFAIK
by dunno7456 2y ago
"to tell crawler to not crawl" which can be ignored AFAIK
- schroeding 2y agoIt can be ignored (it's the equivalent to a "keep out" sign on a lawn), but I very much doubt Google et al. (Edit: Oops, Bing et al.) will actually ignore it.
- Anarch157a 2y agoThe article says Google is paying Reddit to get the data directly from their firehose API, so they wont even bother crawling the public website.
- rbetts 2y agoI wonder if these indexing deals will become more antitrust evidence.
- The_Colonel 2y agoI wonder how much they pay. Reddit profits a lot from showing up on the top for many search queries. I very often do "whatever I'm looking for reddit" (for e.g. product reviews), since the reddit results often provide higher quality information than normal results.
- toomuchtodo 2y ago$60M/year https://www.reuters.com/technology/reddit-ai-content-licensing-deal-with-google-sources-say-2024-02-22/ https://www.reuters.com/technology/reddit-ai-content-licensi...
- dzhiurgis 2y agoGoogle sometimes ignore it when it makes sense (ie big bank accidentally adds login page to ignore) or to check for spam activity (in which case google doesn't use their bot user agent)