3 ms·
That will depend massively on your site structure. (With enough pages Bingbot will do a magnitude more requests than other crawlers) Overall "lower % than your
by viraptor 6y ago
That will depend massively on your site structure. (With enough pages Bingbot will do a magnitude more requests than other crawlers) Overall "lower % than your tooling tells you" - for example cookiebot runs crawlers with both proper user agents and "totally a normal Chrome user" from another network. (Not sure if misconfigured or on purpose)
- Semaphor 6y agoWhen doing some stuff with automated Google Admanager Ads (that required frequent updates), we realized that google is checking every link. At least once per update. Usually more often. With no proper user agent identifying it as a bot. Our solution was to redirect all links through our server and completely block every known google IP-range during the redirect.
- chrisweekly 6y agoI meant net-wide, not for their particular endpoints.