3 ms·
One quick way to detect bots is to look for a GET request of the robots.txt file
by don_draper 14y ago
One quick way to detect bots is to look for a GET request of the robots.txt file
- freeall 14y agoSort of explained already, but "bad" bots would never go look for that file. And a good bot probably already identifies itself the the request, so no need to look through robots.txt.
- fsniper 14y agoThat's it. A "bad bot" would not check robots.txt, but a legitimate user would check it. So looking for the software not checking robots.txt combined with user agent matching for good bots, you would have a good matching ratio.