4 ms·

Google Webmaster Tools allows you control crawling or just a simple robots.txt.

• bonestamp2 12y ago
  He mentioned in the article why neither of those suffice.