2 ms·

IA actually has technical and moral reasons to ignore robots.txt. Namely, they want to circumvent this stuff because their goal is to archive EVERYTHING.

• amarcheschi 2y ago
  I also don't think they hit servers repeatedly so much
• prinny_ 2y ago
  Isn’t this a weak argument? OpenAI could also say their goal is to learn everything, feed it to AI, advance humanity etc etc.
  • compootr 2y ago
    OAI is using others' work to resell it in models. IA uses it to presrrve the history of the web there is a case to be made about the value of the traffic you'll get from oai search though...
    • deleted 2y ago
      [deleted]
  • SR2Z 2y ago
    It does depend a lot on how you feel about IA's integrity :P