I'm sick of it too. In 2023-01 alone I have had 9,000 different IPs from Microsoft's ASN8075 crawling one of my sites with these "normal looking" user agents. P

I'm sick of it too. In 2023-01 alone I have had 9,000 different IPs from Microsoft's ASN8075 crawling one of my sites with these "normal looking" user agents. Poring over the logs to see why your server is on fire takes non-trivial amounts of time. If I didn't have a ton of other stuff to do I'd say it was kinda fun, but I'm freaking fed up. Just yesterday I put all their networks into an nginx geo map: geo $limit_bots_ip { # requests with an empty key are not evaluated by limit_req # see: http://nginx.org/en/docs/http/ngx_http_limit_req_module.html default ''; 157.55.39.0/24 'bot'; 207.46.13.0/24 'bot'; 40.77.167.0/24 'bot'; 13.66.139.0/24 'bot'; ... } Any request from these networks gets classified as a bot, which is then used as the key for a rate limit limit_req_zone $limit_bots_ip zone=badbots_ip:1m rate=1r/m; It's incredible the amount of resources these companies have. I'm just one guy trying to keep a few dozen web servers up.