3 ms·

If they follow robots.txt, OpenAI also has a bot blocking + data gathering problem too: https://x.com/AznWeng/status/1777688628308681000 https://x.com/AznWeng/s

If they follow robots.txt, OpenAI also has a bot blocking + data gathering problem too: https://x.com/AznWeng/status/1777688628308681000 https://x.com/AznWeng/status/1777688628308681000 11% of the top 100K websites already block their crawler, more than all their competitors (Google, FB, Anthropic, Perplexity) combined

Jordan-117 3y ago
It's not just a problem for training, but the end user, too. There are so many times that I've tried to ask a question or request a summary for a long article only to be told it can't read it itself, so you have to copy-paste the text into the chat. Given the non-binding nature of robots.txt and the way they seem comfortable with vacuuming up public data in other contexts, I'm surprised they allow it to be such an obstacle for the user experience.
- lobsterthief 3y ago
  That’s the whole point. The site owner doesn’t want their information included in ChatGPT—they want you going to their website to view it instead. It’s functioning exactly as designed.
  - fragmede 3y ago
    If my web browser's extension "visits" the site and dumps it into ChatGPT for me to read its summarization of the site, what has been gained by the website operator?
    - rsolva 3y ago
      Added friction. That is all a website owner can hope to achieve.
    - pjc50 3y ago
      Nothing. This is why these things - search engines, AI crawlers, even adblock and video downloaders - exist in a slightly adversarial/parasitic relationship with the sites that provide their content to which they provide nothing back (or negative, if you cost them a page load without incurring an ad view). I use adblock all the time but I'm very aware that it can only succeed as long as it doesn't win.
  - jpambrun 3y ago
    It's a stretch to expect a human initiated action to abide by robot.txt. Also, once you click on a link in chrome it's pretty much all robot parsed and rendered from there as well..
    - colonelspace 3y ago
      At bottom, all robot actions are human initiated.
      - s3p 3y ago
        Exactly, so they should ignore robots.txt !
  - Zambyte 3y ago
    I would say robots.txt is meant to filter access for interactions initiated by an automated process (ie automatic crawling). Since the interaction to request a site with a language model is manual (a human request) it doesn't make sense to me that it is used to block that request. If you want to block information you provide from going through ClosedAI servers, block their IPs instead of using robots.txt.

Jordan-117 3y ago

It's not just a problem for training, but the end user, too. There are so many times that I've tried to ask a question or request a summary for a long article only to be told it can't read it itself, so you have to copy-paste the text into the chat. Given the non-binding nature of robots.txt and the way they seem comfortable with vacuuming up public data in other contexts, I'm surprised they allow it to be such an obstacle for the user experience.

lobsterthief 3y ago
That’s the whole point. The site owner doesn’t want their information included in ChatGPT—they want you going to their website to view it instead. It’s functioning exactly as designed.
- fragmede 3y ago
  If my web browser's extension "visits" the site and dumps it into ChatGPT for me to read its summarization of the site, what has been gained by the website operator?
  - rsolva 3y ago
    Added friction. That is all a website owner can hope to achieve.
  - pjc50 3y ago
    Nothing. This is why these things - search engines, AI crawlers, even adblock and video downloaders - exist in a slightly adversarial/parasitic relationship with the sites that provide their content to which they provide nothing back (or negative, if you cost them a page load without incurring an ad view). I use adblock all the time but I'm very aware that it can only succeed as long as it doesn't win.
- jpambrun 3y ago
  It's a stretch to expect a human initiated action to abide by robot.txt. Also, once you click on a link in chrome it's pretty much all robot parsed and rendered from there as well..
  - colonelspace 3y ago
    At bottom, all robot actions are human initiated.
    - s3p 3y ago
      Exactly, so they should ignore robots.txt !
- Zambyte 3y ago
  I would say robots.txt is meant to filter access for interactions initiated by an automated process (ie automatic crawling). Since the interaction to request a site with a language model is manual (a human request) it doesn't make sense to me that it is used to block that request. If you want to block information you provide from going through ClosedAI servers, block their IPs instead of using robots.txt.