5 ms·

robots.txt bro https://developers.openai.com/api/docs/bots/ https://developers.openai.com/api/docs/bots/

• tomalbrc 7mo ago
  "bro": https://www.businessinsider.com/openai-anthropic-ai-ignore-rule-scraping-web-contect-robotstxt?op=1 https://www.businessinsider.com/openai-anthropic-ai-ignore-r...
  • wilg 7mo ago
    should be pretty easy to test and not rely on an anonymous source from a weird analytics company via business insider. are these bots actually from openai or are they just using their user agent? are they coming from openai ip ranges? etc. https://openai.com/gptbot.json https://openai.com/gptbot.json
    • ImPostingOnHN 7mo ago
      > should be pretty easy to test I look forward to your results, whether or not they disprove the article.
    • stefanka 7mo ago
      Are all of OpenAI’s ip ranges known?
  • stefanka 7mo ago
    The article is from 2024. Is this still happening?
    • ImPostingOnHN 7mo ago
      Do we have any evidence they started complying? If not, we can conclude they did not, until such evidence shows up.
      • stefanka 7mo ago
        I’m genuinely curious to know whether there was a change in behavior especially after OpenAI informed about how to prevent scraping (robot.txt, etc.).
        • ImPostingOnHN 7mo ago
          I am as well. Like, is there any evidence of a change, or can we assume nothing changed?