3 ms·
We (kingandmcgaw.com) have seen plenty of examples that suggest Googlebot tries to discover pages that might exist (as distinct from crawling interlinking pages
by anarchitect 11y ago
We (kingandmcgaw.com) have seen plenty of examples that suggest Googlebot tries to discover pages that might exist (as distinct from crawling interlinking pages).
This includes navigating the URL structure (in finding blah.com/one/two/three.html there may be attempts to /one/two and /one) and password-protected admins which are not linked to from anywhere (we suspect Analytics or toolbars are telling Google these pages exist). As a result, Googlebot generates a bunch of false positives in our error logs.