3 ms·
No, but you can identify and block them by other means, such as redirecting bots to an empty page on your web server or blocking them from your router.
by austin-cheney 2y ago
No, but you can identify and block them by other means, such as redirecting bots to an empty page on your web server or blocking them from your router.
- TZubiri 2y agobut how do you identify them?
- knowaveragejoe 2y agoThere's various tricks that can help narrow the criteria, but one that seems particularly good is to put paths no human user would ever find or traverse in robots.txt disallow. Ban anything that visits that path.