3 ms·
Of course it should. But a rule that says you're banned because you're too open with search engines is insane. robots.txt should have nothing to do with it.
by mapgrep 15y ago
Of course it should. But a rule that says you're banned because you're too open with search engines is insane. robots.txt should have nothing to do with it.