AI companies have chosen to either ignore robots.txt entirely or they just rename their crawler so it technically doesn't apply to them. [0] https://www.busine

AI companies have chosen to either ignore robots.txt entirely or they just rename their crawler so it technically doesn't apply to them. [0] https://www.businessinsider.com/openai-anthropic-ai-ignore-rule-scraping-web-contect-robotstxt https://www.businessinsider.com/openai-anthropic-ai-ignore-r... [1] https://www.404media.co/email/87e0e07a-7d24-4788-b417-1821b40c8c1d/?ref=daily-stories-newsletter https://www.404media.co/email/87e0e07a-7d24-4788-b417-1821b4...