3 ms·
In your nginx.conf, http block, add include /etc/nginx/useragent.rules; In /etc/nginx/useragent.rules map $http_user_agent $badagent { defaul
by otherme123 2y ago
In your nginx.conf, http block, add
include /etc/nginx/useragent.rules;
In /etc/nginx/useragent.rules
map $http_user_agent $badagent {
default 0;
~facebook 1;
[...]
~PerplexityBot 1;
}
In your site.conf, server block, add
if ($badagent) {
return 444;
}
- jraph 2y agoAh, nice, way better than my string of ifs.
- nilsherzig 2y agoAnyone knows of a crowd sourced list of these user agents? With the current state of AI startups it will be hard to keep this up to date by myself
- Tepix 2y agoIdeally i would like these crawlers to access /robots.txt but nothing else. Only if they ignore robots.txt the access rules will stop them.
- jraph 2y agoYou can probably write a specific location block for robots.txt which will have a higher priority. See also https://stackoverflow.com/questions/5238377/nginx-location-priority https://stackoverflow.com/questions/5238377/nginx-location-p...