4 ms·
So time, then for a robots.txt with User-agent: Googlebot Disallow: / ? (Of course one would have to find an exhaustive list of all the web-spiders they
by mikro2nd 6y ago
So time, then for a robots.txt with
User-agent: Googlebot
Disallow: /
?
(Of course one would have to find an exhaustive list of all the web-spiders they have and their UA strings, but you get my point...)
- shirakawasuna 6y agoThey could just ignore it, right? Or do any new rules actually require honoring robots.txt?
- occamrazor 6y agoAll major search engines follow robots.txt