4 ms·
How do you filter out crawlers, bots, pre-fetchers, etc.? I've personally never even tried using server-side analytics because I assumed there would be so much
by strommen 11y ago
How do you filter out crawlers, bots, pre-fetchers, etc.?
I've personally never even tried using server-side analytics because I assumed there would be so much noise it wouldn't be worth the trouble.
- TillE 11y agoEvery well-behaved bot (ie, almost all of them) has its own user agent string. You can just grep -v those out.
- mjn 11y agoI wish that were true, but I'd estimate that less than half of the bot hits on my website properly identify themselves as bots. Most of the crawlers running on DigitalOcean/AWS/etc. seem to use a user-agent string lifted from one of the common browsers.
- kuschku 11y agoCheck for the order of the headers and the TLS protocols it claims to support – those are useful for identification.
- th0ma5 11y agoI must be getting old, never? There's a lot of techniques like doings stats on blank pixels and such that have been around for a long time. A lot of ad blocking techniques start at the DNS of the hosted resource, so having a single domain for everything (if practical) can present some reliable information.