43 ms·

have openai's crawlers really paid attention to people's robots.txts?

• simonw 3y ago
  Yes. There's even a project tracking which newspapers are blocking them here: https://palewi.re/docs/news-homepages/openai-gptbot-robotstxt.html https://palewi.re/docs/news-homepages/openai-gptbot-robotstx...
  • input_sh 3y ago
    Oh wow, getting 43.6% of newspapers to do anything collectively over a brief period of time is... definitely not an easy achievement.
    • solardev 3y ago
      Heh, they probably don't want a repeat of Google News
    • is_true 3y ago
      USA today has a lot of newspapers for example. Not sure what % of that list belongs to them nor what's the source.
• theshackleford 3y ago
  It would appear so from the significant network of sites I am responsible for that have put such robots.txt changes into place.