3 ms·
map $http_user_agent $uatype { default 'user'; ~*(googlebot|bingbot) 'good_bot'; ~*(nastybot|somebadscraper) 'ba
by reactordev 8mo ago
map $http_user_agent $uatype {
default 'user';
~*(googlebot|bingbot) 'good_bot';
~*(nastybot|somebadscraper) 'bad_bot';
}
You can also do something like this to rate limit instead of by IP address. Making all ‘bad_bots’ limited but not ‘good_bots’.
I’m not dismissing the difficulty of the problem but there are multiple vectors that can identify these ‘bad_bots’.