4 ms·
The net is full of unidentified bots scraping content or looking for vulnerabilities (contact forms , wordpress logins, etc). On many occasions I had traffic is
by elondaits 7y ago
The net is full of unidentified bots scraping content or looking for vulnerabilities (contact forms , wordpress logins, etc). On many occasions I had traffic issues and had to check logs and these were very hard to block, because they ignore the robots file, don’t advertise themselves in the User Agent and use a large pool of IPs.
- luckylion 7y agoI don't know why this comment is downvoted, it mirrors my experience. I'm responsible for a few domestic high traffic websites and have done some analysis from log files to find suspicious traffic, i.e. user agents saying they are chrome but not loading images or css files, having many page views (i.e. 50 where our average user has 2) etc. It wasn't foolproof but the false-positives where < 10% in my random checks. These bots made up ~10-15% of page views.