It doesn't belong in the guidelines but it is described in the relevant section of the Help Center: http://support.google.com/webmasters/bin/answer.py?hl=en&an

It doesn't belong in the guidelines but it is described in the relevant section of the Help Center: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=35147 http://support.google.com/webmasters/bin/answer.py?hl=en&#38... In summary: If for any reason we cannot reach the robots.txt due to an error (e.g a firewall blocking Googlebot or a 5xx error code when fetching) Googlebot stops its crawling and it's reported in Webmaster Tools as a crawl error. That Help Center article above is about the error message shown in Webmaster Tools. Given that you said you did not see errors being reported, That suggests there was something else going on. If you need more help, our forums are a great place to ask.