3 ms·
AI crawler wars threaten to make the web more closed for everyone
- bediger4000 2y agoI think this article underestimates the proportion of crawler traffic. Big companies like Google, Apple, and some grey area companies like Ahrefs, Bytedance, and Yandex are easy to count. But there's a ton of probably automated GET requests that use user agents typical of human driven browsers. A real browser will ask for images, JavaScript, CSS files after a GET of an HTML file, these probably automated GETs don't have those linked to GETs. I strongly believe that we f you could count the probably automated requests accurately, human driven requests would be a very low proportion of all requests, less than 30%.