I have found this set of htaccess rules to be extremely useful in blocking bad crawling behavior: https://github.com/bluedragonz/bad-bot-blocker https://github

I have found this set of htaccess rules to be extremely useful in blocking bad crawling behavior: https://github.com/bluedragonz/bad-bot-blocker https://github.com/bluedragonz/bad-bot-blocker