4 ms·

Will blocking User-agent: GPTBot in robots.txt work for this too?

judge2020 3y ago
This time it's not scraping the internet (like a robot) but actually acting as a direct user agent for the human typing their prompt, so I wouldn't be against them ignoring robots.txt.
- JohnFen 3y ago
  It's not a crawler, but it seems likely that OpenAI would take the point of view that as long it's going to a website, it may as well keep a cached copy to use for training later.
- stingraycharles 3y ago
  That makes no sense. It’s still automated and not an actual human, and it’s still distilling information from the internet.
  - rolisz 3y ago
    Your browser is also automating things, you're not resolving DNS and writing all the http requests by hand.
    - worthless-trash 3y ago
      I have found the next topic for my blog post. Challenge accepted!
    - jtbayly 3y ago
      So because something is automated somewhere, nobody has to follow robots.TXT anymore?
      - Spivak 3y ago
        You have it backwards, when a human initiates the request, like clicking on a link, your browser (rightfully) will ignore robots.txt. Unattended requests like scrapers do respect robots.txt. This is a case where ChatGPT is acting more like a browser with a funny way of displaying the final result. Each request is initiated by a human so it would likely be reasonable to ignore it in this case.
- luke-stanley 3y ago
  I thought the same but OpenAI may train on user chats by default. Maybe if training's off, they could ignore robots.txt, or they could flag the content to be skipped.
- setuid9002 3y ago
  Wait, so a simple web scraper script has to comply with robots.txt. But if I want to completely ignore the robots.txt, I only have to make my script more complicated (ChatGPT)?
  - seanthemon 3y ago
    And make it complicated enough (a human) and no .txt can stop them!
  - kelseyfrog 3y ago
    Yeah, it's like one weird trick for web scraping.
  - pixl97 3y ago
    I'd like to consider this a difference between script action and user action. For example if you make a web page a user pulls up that calls another webpage, is that a user action, a script action, a mix of both? I personally would consider it a user action.
  - JohnFen 3y ago
    Nobody actually has to comply with robots.txt.
- karaterobot 3y ago
  My understanding is that the philosophy behind robots.txt is owners not wanting their content automatically included in someone else's product, if not duplicated and recorded wholesale. The important idea seems to be ownership, not the ability to browse. If OpenAI had two agents, one with no memory, and one with a memory, that would be better: you could disallow ChaptGPT-storage and allow ChatGPT-user, for example. Barring that, I'd be afraid allowing ChatGPT access to my website means my website is now part of the ChatGPT corpus.
  - spott 3y ago
    https://platform.openai.com/docs/plugins/bot/chatgpt-user-and-gptbot https://platform.openai.com/docs/plugins/bot/chatgpt-user-an... There are two agents, but apparently you can't allow one but disallow another.
  - JohnFen 3y ago
    > My understanding is that the philosophy behind robots.txt is owners not wanting their content automatically included in someone else's product Not really. That use case is done, of course, but the primary purpose of robots.txt is to help crawlers by indicating what parts of the website are appropriate to be searched and what parts aren't. Robots.txt is not intended primarily as a means to defend a site against crawlers. That's why it relies on the goodwill of crawlers to work.
  - tomjen3 3y ago
    As I recall it it was also to block access to scripts, so it wouldn't e.g get stuck pulling pointless stuff out of your local search page.
simonw 3y ago
It uses a different user agent, but yeah it still obeys robots.txt. https://platform.openai.com/docs/plugins/bot https://platform.openai.com/docs/plugins/bot User-agent: ChatGPT-User Disallow: /

judge2020 3y ago

This time it's not scraping the internet (like a robot) but actually acting as a direct user agent for the human typing their prompt, so I wouldn't be against them ignoring robots.txt.

JohnFen 3y ago
It's not a crawler, but it seems likely that OpenAI would take the point of view that as long it's going to a website, it may as well keep a cached copy to use for training later.
stingraycharles 3y ago
That makes no sense. It’s still automated and not an actual human, and it’s still distilling information from the internet.
- rolisz 3y ago
  Your browser is also automating things, you're not resolving DNS and writing all the http requests by hand.
  - worthless-trash 3y ago
    I have found the next topic for my blog post. Challenge accepted!
  - jtbayly 3y ago
    So because something is automated somewhere, nobody has to follow robots.TXT anymore?
    - Spivak 3y ago
      You have it backwards, when a human initiates the request, like clicking on a link, your browser (rightfully) will ignore robots.txt. Unattended requests like scrapers do respect robots.txt. This is a case where ChatGPT is acting more like a browser with a funny way of displaying the final result. Each request is initiated by a human so it would likely be reasonable to ignore it in this case.
luke-stanley 3y ago
I thought the same but OpenAI may train on user chats by default. Maybe if training's off, they could ignore robots.txt, or they could flag the content to be skipped.
setuid9002 3y ago
Wait, so a simple web scraper script has to comply with robots.txt. But if I want to completely ignore the robots.txt, I only have to make my script more complicated (ChatGPT)?
- seanthemon 3y ago
  And make it complicated enough (a human) and no .txt can stop them!
- kelseyfrog 3y ago
  Yeah, it's like one weird trick for web scraping.
- pixl97 3y ago
  I'd like to consider this a difference between script action and user action. For example if you make a web page a user pulls up that calls another webpage, is that a user action, a script action, a mix of both? I personally would consider it a user action.
- JohnFen 3y ago
  Nobody actually has to comply with robots.txt.
karaterobot 3y ago
My understanding is that the philosophy behind robots.txt is owners not wanting their content automatically included in someone else's product, if not duplicated and recorded wholesale. The important idea seems to be ownership, not the ability to browse. If OpenAI had two agents, one with no memory, and one with a memory, that would be better: you could disallow ChaptGPT-storage and allow ChatGPT-user, for example. Barring that, I'd be afraid allowing ChatGPT access to my website means my website is now part of the ChatGPT corpus.
- spott 3y ago
  https://platform.openai.com/docs/plugins/bot/chatgpt-user-and-gptbot https://platform.openai.com/docs/plugins/bot/chatgpt-user-an... There are two agents, but apparently you can't allow one but disallow another.
- JohnFen 3y ago
  > My understanding is that the philosophy behind robots.txt is owners not wanting their content automatically included in someone else's product Not really. That use case is done, of course, but the primary purpose of robots.txt is to help crawlers by indicating what parts of the website are appropriate to be searched and what parts aren't. Robots.txt is not intended primarily as a means to defend a site against crawlers. That's why it relies on the goodwill of crawlers to work.
- tomjen3 3y ago
  As I recall it it was also to block access to scripts, so it wouldn't e.g get stuck pulling pointless stuff out of your local search page.

simonw 3y ago

It uses a different user agent, but yeah it still obeys robots.txt. https://platform.openai.com/docs/plugins/bot https://platform.openai.com/docs/plugins/bot User-agent: ChatGPT-User Disallow: /