3 ms·
Yeah; care to take a guess as to what % of HTTP traffic is human vs bot? (Given your comment, I suspect your guess is likely to be much closer to reality than
by chrisweekly 6y ago
Yeah; care to take a guess as to what % of HTTP traffic is human vs bot?
(Given your comment, I suspect your guess is likely to be much closer to reality than the vast majority of even the relatively clueful people here on HN.)
- viraptor 6y agoThat will depend massively on your site structure. (With enough pages Bingbot will do a magnitude more requests than other crawlers) Overall "lower % than your tooling tells you" - for example cookiebot runs crawlers with both proper user agents and "totally a normal Chrome user" from another network. (Not sure if misconfigured or on purpose)
- Semaphor 6y agoWhen doing some stuff with automated Google Admanager Ads (that required frequent updates), we realized that google is checking every link. At least once per update. Usually more often. With no proper user agent identifying it as a bot. Our solution was to redirect all links through our server and completely block every known google IP-range during the redirect.
- chrisweekly 6y agoI meant net-wide, not for their particular endpoints.