3 ms·

The issue is reciprocity as to crawling. There will never be "news at 11" about that.

• tedunangst 10y ago
  So ban the googlebot in robots.txt and tell google you'll let them back in when they let you crawl their site.
  • __b__ 10y ago
    Or allow Googlebot in robots.txt and then crawl Google.