3 ms·
Disallow them in robots.txt. Visit https://[originating https://[originating IP] and opt out. Email abuse@[owner of their IP]. It would be fun to distinguish an
by 0xdeadbeefbabe 2y ago
Disallow them in robots.txt. Visit https://[originating https://[originating IP] and opt out. Email abuse@[owner of their IP]. It would be fun to distinguish an AI crawler from a regular crawler, btw.
- Lariscus 2y agoAI companies have chosen to either ignore robots.txt entirely or they just rename their crawler so it technically doesn't apply to them. [0] https://www.businessinsider.com/openai-anthropic-ai-ignore-rule-scraping-web-contect-robotstxt https://www.businessinsider.com/openai-anthropic-ai-ignore-r... [1] https://www.404media.co/email/87e0e07a-7d24-4788-b417-1821b40c8c1d/?ref=daily-stories-newsletter https://www.404media.co/email/87e0e07a-7d24-4788-b417-1821b4...