31 ms·
AI scraper bots don't respect robots.txt
by cratermoon 2y ago
AI scraper bots don't respect robots.txt
- jsheard 2y agoI think that's the point, you'd use robots.txt to direct Googlebot/Bingbot/etc away from countermeasures that could potentially mess up your SEO. If other bots ignore the signpost clearly saying not to enter the tarpit, that's their own stupid fault.
- reverendsteveii 2y agoThe ones that survive do