4 ms·
The vast majority of the time when I see a complaint like this about robots.txt, it's usually because the site has missed a character here or there, or because
by Matt_Cutts 15y ago
The vast majority of the time when I see a complaint like this about robots.txt, it's usually because the site has missed a character here or there, or because they're not putting a robots.txt file on the right hostname.
Google has a free robots.txt checker that lets you test your robots.txt files. Given a robots.txt file, you can enter specific urls and check whether that url would be blocked or not. Here's a link for more info on that free tool: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449&from=35237&rd=1 http://www.google.com/support/webmasters/bin/answer.py?hl=en...