3 ms·
Odd, I have always seen the google bot respect the robots.txt. If I disallowed it, it disappeared from my traffic. Of course, generally people do not want to d
by jrochkind1 5y ago
Odd, I have always seen the google bot respect the robots.txt. If I disallowed it, it disappeared from my traffic.
Of course, generally people do not want to do this, as they want to appear in google.