Bot activities in web server logs isn't that hard to spot. They often don't use "regular" user agents, they usually only download a page's HTML without getting

Bot activities in web server logs isn't that hard to spot. They often don't use "regular" user agents, they usually only download a page's HTML without getting all the images and media, they often do a GET on /robots.txt before proceeding, they manifest unrealisticly short times between browsing a page and going to the next one, and they often come from known IP subnets. My simple log parsing heuristic can already remove 90% of the bot traffic from the statistics.