"Time-between-requests would seem reasonable except that bots ... I suspect you may get some false positives." Solid point. "Perhaps you could operate this in

"Time-between-requests would seem reasonable except that bots ... I suspect you may get some false positives." Solid point. "Perhaps you could operate this in a log-only passive mode for a while to log what would have been blocked?" I can and will. This seems to be the safest way forward. "The same applies to malicious bots. They can be crawling hundreds of sites at the same time. There is no way to guarantee your site gets the same threads and resources by the bots. Some of them are using pools of LTE modems so even their IP's will be spread across a large CIDR block." This is very true. Best case scenario: you block all malicious bots... but then they evolve. I'm viewing this from a resource efficiency perspective. I want to limit the cost of automated hits. "That said, if you specify a crawl delay of 2 seconds in robots.txt and the bots do not obey this then perhaps it may be worth blocking them." Which makes this a great idea. It's simple and dichotomous. All desirable bots will respect this declaration. "Allowing HTTP/1.1 for the bots that specify the google user-agent will not accommodate their bots that pretend to be Android phones and may result in Google flagging your site as potentially malicious. Some malicious bot tools will also use their user-agent in attempt to bypass authentication on some sites." Ah, I just realized Googlebot and these "android bots" use 1.1. Wouldn't be a problem otherwise. I'll leave this idea alone for now. Never anger the SERP gods. :D Edit: It seems Googlebot does not accept the crawl-delay directive. Instead, it needs to be set in Search Console every 90 days. https://support.google.com/webmasters/answer/48620?hl=en https://support.google.com/webmasters/answer/48620?hl=en