3 ms·
My understanding is that the philosophy behind robots.txt is owners not wanting their content automatically included in someone else's product, if not duplicate
by karaterobot 3y ago
My understanding is that the philosophy behind robots.txt is owners not wanting their content automatically included in someone else's product, if not duplicated and recorded wholesale. The important idea seems to be ownership, not the ability to browse. If OpenAI had two agents, one with no memory, and one with a memory, that would be better: you could disallow ChaptGPT-storage and allow ChatGPT-user, for example. Barring that, I'd be afraid allowing ChatGPT access to my website means my website is now part of the ChatGPT corpus.
- spott 3y agohttps://platform.openai.com/docs/plugins/bot/chatgpt-user-and-gptbot https://platform.openai.com/docs/plugins/bot/chatgpt-user-an... There are two agents, but apparently you can't allow one but disallow another.
- JohnFen 3y ago> My understanding is that the philosophy behind robots.txt is owners not wanting their content automatically included in someone else's product Not really. That use case is done, of course, but the primary purpose of robots.txt is to help crawlers by indicating what parts of the website are appropriate to be searched and what parts aren't. Robots.txt is not intended primarily as a means to defend a site against crawlers. That's why it relies on the goodwill of crawlers to work.
- tomjen3 3y agoAs I recall it it was also to block access to scripts, so it wouldn't e.g get stuck pulling pointless stuff out of your local search page.