4 ms·

Is there a complete list of ip addresses openai uses for scraping? I suppose they dont honour robots.txt, despite claiming to do so, and this may be one way to

Is there a complete list of ip addresses openai uses for scraping? I suppose they dont honour robots.txt, despite claiming to do so, and this may be one way to reliably block them.

londons_explore 3y ago
I've seen no suggestion of them not honouring robots.txt... I suspect that any case of them not honoring it is probably due to your site content being available on archive.org or common crawl or some other service.
- kelseyfrog 3y ago
  Is there an open license or ToS that would disallow OpenAI access?
  - gumballindie 3y ago
    Apparently openai and the rest of defendants claim that if it’s on the internet it’s “fair use”. Meaning they’ll do all they can tk steal your work regardless of licensing or robots.txt rules.
    - kelseyfrog 3y ago
      That's a funny argument. It's a lot like saying, "if it's on Github, I can ignore your LICENSE file" or "if it's on DA I can ignore your CC-NC license." Both of these seem technically enforceable (if not expensive) so I'm not quite understanding what OpenAI is grounding their justification in. It seems much more likely that OpenAI stands to make a lot of money by arguing that it's fair use and then back filling in an argument rather than making one from first principles.
      - gumballindie 3y ago
        A”I” depends on data. The more the better. That’s why a lot of these people push for a disregard of people’s property (digital content, behaviour data, etc). Problem is in stealing all of it they will demotivate people from creating quality content and even alienate them from the dead internet. OpenAI leads the cult.
JohnFen 3y ago
If your concern is that you don't want the contents of your site to be used to train AI, then OpenAI is not the only entity you need to protect against. Other crawlers may not respect robots.txt, and blocking by IP address is just a game of whack-a-mole that you can't win. This is why I took down some of my sites, and put a login in front of the rest. Until I have some solid means of defense, I can't think of any other effective approach.

Is there a complete list of ip addresses openai uses for scraping? I suppose they dont honour robots.txt, despite claiming to do so, and this may be one way to reliably block them.

londons_explore 3y ago

I've seen no suggestion of them not honouring robots.txt... I suspect that any case of them not honoring it is probably due to your site content being available on archive.org or common crawl or some other service.

kelseyfrog 3y ago
Is there an open license or ToS that would disallow OpenAI access?
- gumballindie 3y ago
  Apparently openai and the rest of defendants claim that if it’s on the internet it’s “fair use”. Meaning they’ll do all they can tk steal your work regardless of licensing or robots.txt rules.
  - kelseyfrog 3y ago
    That's a funny argument. It's a lot like saying, "if it's on Github, I can ignore your LICENSE file" or "if it's on DA I can ignore your CC-NC license." Both of these seem technically enforceable (if not expensive) so I'm not quite understanding what OpenAI is grounding their justification in. It seems much more likely that OpenAI stands to make a lot of money by arguing that it's fair use and then back filling in an argument rather than making one from first principles.
    - gumballindie 3y ago
      A”I” depends on data. The more the better. That’s why a lot of these people push for a disregard of people’s property (digital content, behaviour data, etc). Problem is in stealing all of it they will demotivate people from creating quality content and even alienate them from the dead internet. OpenAI leads the cult.

JohnFen 3y ago

If your concern is that you don't want the contents of your site to be used to train AI, then OpenAI is not the only entity you need to protect against. Other crawlers may not respect robots.txt, and blocking by IP address is just a game of whack-a-mole that you can't win. This is why I took down some of my sites, and put a login in front of the rest. Until I have some solid means of defense, I can't think of any other effective approach.