4 ms·
> Except the fiist thing openai does is read robots.txt. Then they should see the "Disallow: /" line, which means they shouldn't crawl any links on the page (b
by flutas 3y ago
> Except the fiist thing openai does is read robots.txt.
Then they should see the "Disallow: /" line, which means they shouldn't crawl any links on the page (because even the homepage is disallowed). Which means they wouldn't follow any of the links to other subdomains.
- niutech 3y agoThis robots.txt has Disallow rule commented out: # buzz off #User-agent: GPTBot #Disallow: /