4 ms·
It's time to brush up robots.txt
- cratermoon 3y agoTo the extent that robots.txt still works, see Block the Bots that Feed “AI” Models by Scraping Your Website: https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/ https://neil-clarke.com/block-the-bots-that-feed-ai-models-b... You can also block user agents directly, for example in nginx. https://www.xmodulo.com/block-specific-user-agents-nginx-web-server.html https://www.xmodulo.com/block-specific-user-agents-nginx-web... The UAs I'm aware of: CCBot ChatGPT-User GPTBot Does anyone know of a resource that tracks these crawlers' UA strings?