4 ms·
Do Linux based spiders and crawlers count? If so, has that skewed the results in a meaningful way?
by daniel-levin 14y ago
Do Linux based spiders and crawlers count? If so, has that skewed the results in a meaningful way?
- padraigm 14y agoWhether they count or not, I would expect to see "regular" users to be orders of magnitude more common than web crawlers.
- Flimm 14y agoBut web crawlers generate orders of magnitude more traffic than regular users.
- dredmorbius 14y agoTrue, but if you're tracking users, you'll normalize for that (tracking cookies, bot exclusion, IP, etc.). It's somewhat nontrivial, but most web analytics packages will give you a decent first cut.
- pyvek 14y agoUnless the web crawlers are intentionally spoofing their user agent to identify as browsers, I don't think that they are counted.