Something like User-agent: * Disallow: / User-agent: Googlebot Allow: / This can be found on many /robots.txt files. I imagine that some of them wi

Something like User-agent: * Disallow: / User-agent: Googlebot Allow: / This can be found on many /robots.txt files. I imagine that some of them will try to enforce this access policy. My point is that equal access for all has never existed. (Not that it would be realistic to expect it.)