3 ms·

> Hey! I'm Nick, and I work on Integrity at OpenAI. These checks are part of how we protect our first-party products from abuse like bots, scraping, fraud, and

by vkou 7mo ago

> Hey! I'm Nick, and I work on Integrity at OpenAI. These checks are part of how we protect our first-party products from abuse like bots, scraping, fraud, and other attempts to misuse the platform. How can first-party products protect themselves from abuse by OpenAI's bots and scraping?

mystraline 7mo ago
This is a completely in-scope question. How do we defend against your scraping, OpenAI? I dont want any of my content scraped or seen by you all. Frankly, fuck you all for thinking my content is owned by you.
- wilg 7mo ago
  robots.txt bro https://developers.openai.com/api/docs/bots/ https://developers.openai.com/api/docs/bots/
  - tomalbrc 7mo ago
    "bro": https://www.businessinsider.com/openai-anthropic-ai-ignore-rule-scraping-web-contect-robotstxt?op=1 https://www.businessinsider.com/openai-anthropic-ai-ignore-r...
    - wilg 7mo ago
      should be pretty easy to test and not rely on an anonymous source from a weird analytics company via business insider. are these bots actually from openai or are they just using their user agent? are they coming from openai ip ranges? etc. https://openai.com/gptbot.json https://openai.com/gptbot.json
      - ImPostingOnHN 7mo ago
        > should be pretty easy to test I look forward to your results, whether or not they disprove the article.
      - stefanka 7mo ago
        Are all of OpenAI’s ip ranges known?
    - stefanka 7mo ago
      The article is from 2024. Is this still happening?
      - ImPostingOnHN 7mo ago
        Do we have any evidence they started complying? If not, we can conclude they did not, until such evidence shows up.
        stefanka 7mo ago
        I’m genuinely curious to know whether there was a change in behavior especially after OpenAI informed about how to prevent scraping (robot.txt, etc.).
        ImPostingOnHN 7mo ago
        I am as well. Like, is there any evidence of a change, or can we assume nothing changed?
- CableNinja 7mo ago
  I use nginx conditionals and useragent checking, then respond with 418 or 410. Probably too late now but my list needs updating
- tedsanders 7mo ago
  It's documented here: https://developers.openai.com/api/docs/bots https://developers.openai.com/api/docs/bots
  - subscribed 7mo ago
    - which one is to stop you from hammering small servers with hundreds simultaneous connections? - why don't you just respect existing robots.txt that apply to you already? - does every LLM scraper seriously think the onus to opt out from the EVERY SINGLE SCRAPER is on the webmasters/owners?
    - DrewADesign 6mo ago
      >* - why don't you just respect existing robots.txt that apply to you already?* >* - does every LLM scraper seriously think the onus to opt out from the EVERY SINGLE SCRAPER is on the webmasters/owners?* I see value in asking explicitly, but let’s be real here: you know the answers. They feel entitled to vacuum up everything visible, any way they feel like, for some ratio of profit to thinking they’re reshaping humanity for the better. The opt-outs are ceremonial procedures to convince people (including themselves,) that people are actually volunteering this data. They don’t give a shit about you, your creative output, you life’s work, your server performance and bandwidth, or anything else beyond stuffing the great data pig.

by vkou 7mo ago

mystraline 7mo ago

This is a completely in-scope question. How do we defend against your scraping, OpenAI? I dont want any of my content scraped or seen by you all. Frankly, fuck you all for thinking my content is owned by you.

wilg 7mo ago
robots.txt bro https://developers.openai.com/api/docs/bots/ https://developers.openai.com/api/docs/bots/
- tomalbrc 7mo ago
  "bro": https://www.businessinsider.com/openai-anthropic-ai-ignore-rule-scraping-web-contect-robotstxt?op=1 https://www.businessinsider.com/openai-anthropic-ai-ignore-r...
  - wilg 7mo ago
    should be pretty easy to test and not rely on an anonymous source from a weird analytics company via business insider. are these bots actually from openai or are they just using their user agent? are they coming from openai ip ranges? etc. https://openai.com/gptbot.json https://openai.com/gptbot.json
    - ImPostingOnHN 7mo ago
      > should be pretty easy to test I look forward to your results, whether or not they disprove the article.
    - stefanka 7mo ago
      Are all of OpenAI’s ip ranges known?
  - stefanka 7mo ago
    The article is from 2024. Is this still happening?
    - ImPostingOnHN 7mo ago
      Do we have any evidence they started complying? If not, we can conclude they did not, until such evidence shows up.
      - stefanka 7mo ago
        I’m genuinely curious to know whether there was a change in behavior especially after OpenAI informed about how to prevent scraping (robot.txt, etc.).
        ImPostingOnHN 7mo ago
        I am as well. Like, is there any evidence of a change, or can we assume nothing changed?
CableNinja 7mo ago
I use nginx conditionals and useragent checking, then respond with 418 or 410. Probably too late now but my list needs updating
tedsanders 7mo ago
It's documented here: https://developers.openai.com/api/docs/bots https://developers.openai.com/api/docs/bots
- subscribed 7mo ago
  - which one is to stop you from hammering small servers with hundreds simultaneous connections? - why don't you just respect existing robots.txt that apply to you already? - does every LLM scraper seriously think the onus to opt out from the EVERY SINGLE SCRAPER is on the webmasters/owners?
  - DrewADesign 6mo ago
    >* - why don't you just respect existing robots.txt that apply to you already?* >* - does every LLM scraper seriously think the onus to opt out from the EVERY SINGLE SCRAPER is on the webmasters/owners?* I see value in asking explicitly, but let’s be real here: you know the answers. They feel entitled to vacuum up everything visible, any way they feel like, for some ratio of profit to thinking they’re reshaping humanity for the better. The opt-outs are ceremonial procedures to convince people (including themselves,) that people are actually volunteering this data. They don’t give a shit about you, your creative output, you life’s work, your server performance and bandwidth, or anything else beyond stuffing the great data pig.