3 ms·

Keep in mind that the Internet actually has more bot traffic than human traffic these days. A 50% discrepancy between Google Analytics and access logs could be

by lmkg 6y ago

Keep in mind that the Internet actually has more bot traffic than human traffic these days. A 50% discrepancy between Google Analytics and access logs could be explained entirely by bot traffic with no ad-blockers involved. It could be ad-blockers, though. And I want to emphasize a point you mentioned briefly: it very much depends on your target audience. An anecdote from several years ago, around with IE was still the plurality of traffic on most sites but no longer the majority. Two web properties run by the same company, with different audiences. On one of them, IE did not crack the Top 5 in browsers, and was actually outnumbered by Opera. On another, the entire Top 5 browsers were different versions of IE, and Chrome and Firefox combined were less popular than IE6, IE5.5, and AOL's browser (which apparently was a reskinned IE7). The moral of the story is, what's true for everyone may not be true for you, and vice-versa.

lucb1e 6y ago
The vast majority of bot traffic identifies itself though. Search engines from all over that you've never heard of, but they're all showing accurate user agents. I don't attribute every access by far, of course, but for a small website (with peaks to high HN ranks, so it's not unknown or that badly linked) it's manageable to tail -f the access log and see that the traffic you're getting matches what you expect after e.g. posting a link in a chat group. About 99% of traffic in such a case fall within the categories "human-looking user agent visiting this newly created URL that I just shared in the group" and "random bot traffic".
- mfontani 6y ago
  The vast majority of what I guess is "robotic, but javascript-running" traffic identifies itself as the (headless) browser it most likely uses, i.e. Chrome. When such crawlers use a distributed network of machines to crawl, there's no easy way to identify them as such, other than _guess_ that they're robotic in nature due to them coming from "data centre" like IPs (i.e. AWS, Azure, GCP, waht-have-you). While such requests may well be users using those "data centre" services for, say, a personal VPN... the overall pattern is what gives them away. Not all users look at a page and then start requesting (maybe from a separate IP, mind you, but still from the same provider yet with the same UA) each URL in that page in succession. Crawlers who want to stay under the radar are much, much harder to detect as such nowadays.
  - lucb1e 6y ago
    En masse, with a popular website, sure. But from my limited traffic website where I sometimes look at individual requests by hand and see that it's my friends from residential Dutch IP addresses loading the page, I'm quite sure these are no secret masqueraded internet robots. Hence I conclude that, since much of the traffic (in periods that I look at randomly, the vast majority I never look at) is identifiable as human or as clearly a crawler, such unidentifiable bot traffic is a very tiny percentage. I'm sure it'll be there, lots of people will be interested in what differences a page shows from different vantage points or to robots, but it's probably less than a percent of all robot traffic. Though checking the thread above mine, I'm not exactly sure what I was replying to. Didn't it say something about robot traffic that wasn't identifiable as such? I don't see that anymore.

by lmkg 6y ago

lucb1e 6y ago

The vast majority of bot traffic identifies itself though. Search engines from all over that you've never heard of, but they're all showing accurate user agents. I don't attribute every access by far, of course, but for a small website (with peaks to high HN ranks, so it's not unknown or that badly linked) it's manageable to tail -f the access log and see that the traffic you're getting matches what you expect after e.g. posting a link in a chat group. About 99% of traffic in such a case fall within the categories "human-looking user agent visiting this newly created URL that I just shared in the group" and "random bot traffic".

mfontani 6y ago
The vast majority of what I guess is "robotic, but javascript-running" traffic identifies itself as the (headless) browser it most likely uses, i.e. Chrome. When such crawlers use a distributed network of machines to crawl, there's no easy way to identify them as such, other than _guess_ that they're robotic in nature due to them coming from "data centre" like IPs (i.e. AWS, Azure, GCP, waht-have-you). While such requests may well be users using those "data centre" services for, say, a personal VPN... the overall pattern is what gives them away. Not all users look at a page and then start requesting (maybe from a separate IP, mind you, but still from the same provider yet with the same UA) each URL in that page in succession. Crawlers who want to stay under the radar are much, much harder to detect as such nowadays.
- lucb1e 6y ago
  En masse, with a popular website, sure. But from my limited traffic website where I sometimes look at individual requests by hand and see that it's my friends from residential Dutch IP addresses loading the page, I'm quite sure these are no secret masqueraded internet robots. Hence I conclude that, since much of the traffic (in periods that I look at randomly, the vast majority I never look at) is identifiable as human or as clearly a crawler, such unidentifiable bot traffic is a very tiny percentage. I'm sure it'll be there, lots of people will be interested in what differences a page shows from different vantage points or to robots, but it's probably less than a percent of all robot traffic. Though checking the thread above mine, I'm not exactly sure what I was replying to. Didn't it say something about robot traffic that wasn't identifiable as such? I don't see that anymore.