I use fail2ban for this. I know what my servers run (and it's never PHP, Python microsoft/AD sharepoint etc, because I don't have services in these languages or

I use fail2ban for this. I know what my servers run (and it's never PHP, Python microsoft/AD sharepoint etc, because I don't have services in these languages or systems) so I simply fail2ban everything that matches these broad items. Access "wp-admin"? Ban. Try "cron.php" ban. Looking for "phpmyadmin"? ban. It works reasonably well. These bots can switch to other IPs, or try again after the jailtime is lifted (20 mins in my case). But instead of a bot attempting thousands of endpoints, I get only one attempt. I initially configured this on a backend, that for reasons, had to handle every URL in the (Ruby, so heavy and slow) application layer: a bot trying common endpoints for popular CMSes and services, would put severe load on this application. Fail2ban was already there, blocking SSH, FTP, SMTP and whatnot, so I configured it to also handle these HTTP endpoints.