4 ms·
I imagine it's everything on the web that's been crawled by Bing and hasn't blocked ChatGPT explicitly in its robots.txt file.
by simonw 3y ago
I imagine it's everything on the web that's been crawled by Bing and hasn't blocked ChatGPT explicitly in its robots.txt file.
- willsmith72 3y agohave openai's crawlers really paid attention to people's robots.txts?
- simonw 3y agoYes. There's even a project tracking which newspapers are blocking them here: https://palewi.re/docs/news-homepages/openai-gptbot-robotstxt.html https://palewi.re/docs/news-homepages/openai-gptbot-robotstx...
- input_sh 3y agoOh wow, getting 43.6% of newspapers to do anything collectively over a brief period of time is... definitely not an easy achievement.
- theshackleford 3y agoIt would appear so from the significant network of sites I am responsible for that have put such robots.txt changes into place.