5 ms·
>configured to block anything that had the word 'bot' as part of its user agent string Yep, that'll do it :) Glad you found it and fixed it quickly! One follo
by pierrefar 15y ago
>configured to block anything that had the word 'bot' as part of its user agent string
Yep, that'll do it :) Glad you found it and fixed it quickly!
One follow-up thought for you and anyone in this situation: Set up a Google Webmaster Tools account ( http://www.google.com/webmasters/tools/ http://www.google.com/webmasters/tools/ ) and make reviewing it part of the webmasters' daily routine. In this particular case, the Crawl Errors page in the Diagnostics section would have flagged this problem very quickly.
A couple of pro tips for Webmaster Tools:
1. Be sure to look at the "date detected" column because it is accurate and somehow people miss it.
2. Set up email forwarding for messages Webmaster Tools sends to you: http://www.google.com/support/webmasters/bin/answer.py?answer=140528 http://www.google.com/support/webmasters/bin/answer.py?answe...
- danheberden 15y agoOdd, but the Crawl Errors actually displayed "No known errors" - which is odd, since it was definitely having issues fetching as googlebot. Thanks for the suggestions and finds, though - we'll be looking at it more often for sure! :)
- pierrefar 15y agoThat's usually the case if you've just verified the site for the first time. It takes a few days for the first data to be collected. Because of this we always recommend webmasters verify early (before things break) and monitor constantly.