4 ms·
Sadly I imagine most of the traffic to a server would be bots or bad actors scanning for common vulns. I used AWStats (Another log file analyzer) for many years
by legrande 5y ago
Sadly I imagine most of the traffic to a server would be bots or bad actors scanning for common vulns. I used AWStats (Another log file analyzer) for many years and had to slice roughly 70% percent of my traffic away because most of it was automated.
Most bots were courteous to state they were bots typically using a useragent with `-bot` found in the string. Some used generic browser useragents but were scanning for things like `wp-admin` etc
Most of the genuine human traffic were people on a mobile phone and that was the only heuristic I looked at to determine how many people visited my site. Very few desktop users were present.
- npilk 5y agoThe --ignore-crawlers flag in GoAccess filters out a decent amount of bot traffic. It's not perfect, but it's good enough for a rough estimate of 'real' traffic.
- loloquwowndueo 5y agoIndeed - about 39% of hits to my site come from crawlers. How do I know? Thanks to GoAccess!! ;)
- jonatron 5y agoYou could add $ssl_cipher to your log_format configuration (if nginx), and use that as a TLS fingerprint to find more bots.
- jgalt212 5y agoHere's a tricky one. Common Crawl runs its bots from AWS. AWS has like a jillion IP addresses. How do you tell which traffic are legit Common Crawl bots and which are imposters?
- ComputerGuru 5y agoI mean, they’re all bots one way or the other. The only exception would be a personal VPN running off of AWS, but that’s a bad idea given how many sites block that range.