3 ms·
For the "good" bots which at least respect robots.txt you can use this list to get ahead of them before they pummel your site. https://github.com/ai-robots-txt
by jsheard 2y ago
For the "good" bots which at least respect robots.txt you can use this list to get ahead of them before they pummel your site.
https://github.com/ai-robots-txt/ai.robots.txt https://github.com/ai-robots-txt/ai.robots.txt
There's no easy solution for bad bots which ignore robots.txt and spoof their UA though.
- taikahessu 2y agoThanks, will look into that!
- breakingcups 2y agoSuch as OpenAI, who will ignore robots.txt and change their user agent to evade blocks, apparently[1] 1: https://www.reddit.com/r/selfhosted/comments/1i154h7/openai_not_respecting_robotstxt_and_being_sneaky/ https://www.reddit.com/r/selfhosted/comments/1i154h7/openai_...
- zcase 2y agoFor those looking, this is the best I've found: https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/ https://blog.cloudflare.com/declaring-your-aindependence-blo...
- maeil 2y agoThis seemed to work for some time when it came out but IME no longer does.