3 ms·
I've seen some of these bots take a lot of CPU on my server, especially when browsing my (very small) forgejo instance. I banned them with a 444 error [1] in th
by jraph 2y ago
I've seen some of these bots take a lot of CPU on my server, especially when browsing my (very small) forgejo instance. I banned them with a 444 error [1] in the reverse proxy settings as a temporary measure that became permanent, and then some more from this list [2], but I will consider yours as well, thanks for sharing.
if ($http_user_agent ~ facebook) { return 444; }
if ($http_user_agent ~ Amazonbot) { return 444; }
if ($http_user_agent ~ Bytespider) { return 444; }
if ($http_user_agent ~ GPTBot) { return 444; }
if ($http_user_agent ~ ClaudeBot) { return 444; }
if ($http_user_agent ~ ImagesiftBot) { return 444; }
if ($http_user_agent ~ CCBot) { return 444; }
if ($http_user_agent ~ ChatGPT-User) { return 444; }
if ($http_user_agent ~ omgili) { return 444; }
if ($http_user_agent ~ Diffbot) { return 444; }
if ($http_user_agent ~ Claude-Web) { return 444; }
if ($http_user_agent ~ PerplexityBot) { return 444; }
(edit: see replies to do it in a cleaner way)
[1] https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#nginx https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#ngin...
[2] https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/ https://blog.cloudflare.com/declaring-your-aindependence-blo...
- otherme123 2y agoIn your nginx.conf, http block, add include /etc/nginx/useragent.rules; In /etc/nginx/useragent.rules map $http_user_agent $badagent { default 0; ~facebook 1; [...] ~PerplexityBot 1; } In your site.conf, server block, add if ($badagent) { return 444; }
- jraph 2y agoAh, nice, way better than my string of ifs.
- nilsherzig 2y agoAnyone knows of a crowd sourced list of these user agents? With the current state of AI startups it will be hard to keep this up to date by myself
- Tepix 2y agoIdeally i would like these crawlers to access /robots.txt but nothing else. Only if they ignore robots.txt the access rules will stop them.
- jraph 2y agoYou can probably write a specific location block for robots.txt which will have a higher priority. See also https://stackoverflow.com/questions/5238377/nginx-location-priority https://stackoverflow.com/questions/5238377/nginx-location-p...