They do respect robots.txt (supposedly), but they also introduced a new user agent that nobody would yet have in their robots.txt as part of this feature[1], an

They do respect robots.txt (supposedly), but they also introduced a new user agent that nobody would yet have in their robots.txt as part of this feature[1], and looking at my server logs it's already crawled a bunch of sites. [1] https://platform.openai.com/docs/bots/overview-of-openai-crawlers https://platform.openai.com/docs/bots/overview-of-openai-cra...