4 ms·
The vast majority of what I guess is "robotic, but javascript-running" traffic identifies itself as the (headless) browser it most likely uses, i.e. Chrome. Wh
by mfontani 6y ago
The vast majority of what I guess is "robotic, but javascript-running" traffic identifies itself as the (headless) browser it most likely uses, i.e. Chrome.
When such crawlers use a distributed network of machines to crawl, there's no easy way to identify them as such, other than _guess_ that they're robotic in nature due to them coming from "data centre" like IPs (i.e. AWS, Azure, GCP, waht-have-you).
While such requests may well be users using those "data centre" services for, say, a personal VPN... the overall pattern is what gives them away.
Not all users look at a page and then start requesting (maybe from a separate IP, mind you, but still from the same provider yet with the same UA) each URL in that page in succession.
Crawlers who want to stay under the radar are much, much harder to detect as such nowadays.
- lucb1e 6y agoEn masse, with a popular website, sure. But from my limited traffic website where I sometimes look at individual requests by hand and see that it's my friends from residential Dutch IP addresses loading the page, I'm quite sure these are no secret masqueraded internet robots. Hence I conclude that, since much of the traffic (in periods that I look at randomly, the vast majority I never look at) is identifiable as human or as clearly a crawler, such unidentifiable bot traffic is a very tiny percentage. I'm sure it'll be there, lots of people will be interested in what differences a page shows from different vantage points or to robots, but it's probably less than a percent of all robot traffic. Though checking the thread above mine, I'm not exactly sure what I was replying to. Didn't it say something about robot traffic that wasn't identifiable as such? I don't see that anymore.