2 ms·
Is there a reason for any "/x?" URLs to be indexed? If not, a simple robots.txt would fix the crawler-traffic problem without prejudicially banning any upstart
by gojomo 19y ago
Is there a reason for any "/x?" URLs to be indexed?
If not, a simple robots.txt would fix the crawler-traffic problem without prejudicially banning any upstart search engines:
User-Agent: *
Disallow: /x?