4 ms·
AI crawlers, famous for respecting robots.txt ;)
by mysterydip 3mo ago
AI crawlers, famous for respecting robots.txt ;)
- dannyw 3mo agoThe western AI crawlers are generally very-well behaved. GPTBot and ClaudeBot are nice crawlers and I see them respecting my robots.txt. There are of course plenty of mystery, obfuscated/camouflaged scrapers/crawlers from god knows whom. Thankfully they are easy to spot and ban, although I've definitely thought about deliberating sending them poisoned data.