3 ms·
I've heard that too. Your posting inspired me to check. I see daily accesses from claudebot and gtpbot to /robots.txt on two sites. Also from something called a
by Arnt 2y ago
I've heard that too. Your posting inspired me to check. I see daily accesses from claudebot and gtpbot to /robots.txt on two sites. Also from something called amazonbot, which may be another AI bot?
- osdotsystem 2y agoMe i dropped a bunch of dont allows for different bots, as many as i could find.
- Arnt 2y agoI'm curious — if that's your attitude, why not all bots? You can say "User-Agent: foo\nDisallow:\n\nUser-Agent: *\nDisallow: /" to allow only named bots and reject them by default.