3 ms·
Last I checked, Google's crawler treated 4xx errors on a /robots.txt (which includes unauthorized/not-allowed) the same as 404-not-found. (In most cases, it's a
by gojomo 17y ago
Last I checked, Google's crawler treated 4xx errors on a /robots.txt (which includes unauthorized/not-allowed) the same as 404-not-found. (In most cases, it's a configuration error by the site owner, and for those sites where it's intentional, the operator is savvy enough to block crawler access to other URLs by password or user-agent if desired.)