4 ms·

I believe that a number of AI bots only respect robot.txt entries that explicitly define their static user agent name. They ignore wildcards in user agents. Th

I believe that a number of AI bots only respect robot.txt entries that explicitly define their static user agent name. They ignore wildcards in user agents. That counts as barely imho. I found this out after OpenAI was decimating my site and ignoring the wildcard deny all. I had to add entires specifically for their three bots to get them to stop.

noman-land 2y ago
This is highly annoying and rude. Is there a complete list of all known bots and crawlers?
- jsheard 2y ago
  https://darkvisitors.com/agents https://darkvisitors.com/agents https://github.com/ai-robots-txt/ai.robots.txt https://github.com/ai-robots-txt/ai.robots.txt
joecool1029 2y ago
Even some non-profit ignore it now, Internet Archive stopped respecting it years ago: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
- SR2Z 2y ago
  IA actually has technical and moral reasons to ignore robots.txt. Namely, they want to circumvent this stuff because their goal is to archive EVERYTHING.
  - amarcheschi 2y ago
    I also don't think they hit servers repeatedly so much
  - prinny_ 2y ago
    Isn’t this a weak argument? OpenAI could also say their goal is to learn everything, feed it to AI, advance humanity etc etc.
    - compootr 2y ago
      OAI is using others' work to resell it in models. IA uses it to presrrve the history of the web there is a case to be made about the value of the traffic you'll get from oai search though...
      - deleted 2y ago
        [deleted]
    - SR2Z 2y ago
      It does depend a lot on how you feel about IA's integrity :P
- AnonC 2y ago
  As I recall, this is outdated information. Internet Archive does respect robots.txt and will remove a site from its archive based on robots.txt. I have done this a few years after your linked blog post to get an inconsequential site removed from archive.org.
  - dredmorbius 2y ago
    The most recent notice IA have blogged was in 2017, and there's no indication that the service has reversed course on robots.txt since. <https://blog.archive.org/?s=robots.txt https://blog.archive.org/?s=robots.txt>
deleted 2y ago
[deleted]

noman-land 2y ago

This is highly annoying and rude. Is there a complete list of all known bots and crawlers?

jsheard 2y ago
https://darkvisitors.com/agents https://darkvisitors.com/agents https://github.com/ai-robots-txt/ai.robots.txt https://github.com/ai-robots-txt/ai.robots.txt

joecool1029 2y ago

Even some non-profit ignore it now, Internet Archive stopped respecting it years ago: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

SR2Z 2y ago
IA actually has technical and moral reasons to ignore robots.txt. Namely, they want to circumvent this stuff because their goal is to archive EVERYTHING.
- amarcheschi 2y ago
  I also don't think they hit servers repeatedly so much
- prinny_ 2y ago
  Isn’t this a weak argument? OpenAI could also say their goal is to learn everything, feed it to AI, advance humanity etc etc.
  - compootr 2y ago
    OAI is using others' work to resell it in models. IA uses it to presrrve the history of the web there is a case to be made about the value of the traffic you'll get from oai search though...
    - deleted 2y ago
      [deleted]
  - SR2Z 2y ago
    It does depend a lot on how you feel about IA's integrity :P
AnonC 2y ago
As I recall, this is outdated information. Internet Archive does respect robots.txt and will remove a site from its archive based on robots.txt. I have done this a few years after your linked blog post to get an inconsequential site removed from archive.org.
- dredmorbius 2y ago
  The most recent notice IA have blogged was in 2017, and there's no indication that the service has reversed course on robots.txt since. <https://blog.archive.org/?s=robots.txt https://blog.archive.org/?s=robots.txt>

deleted 2y ago

[deleted]