3 ms·
Do you use Google webmaster tools? They should show potential reasons on there and which pages are currently indexed.
by rcavezza 14y ago
Do you use Google webmaster tools? They should show potential reasons on there and which pages are currently indexed.
- Frozenlock 14y agoThe reason shown in the crawl error is "Robots.txt fetch". This isn't very surprising as 'fetch as Google' returns a page unavailable for everything. (It's my understanding that a 404 on robots.txt is OK)
- 27182818284 14y agoIn the last few days I've seen errors with our robots.txt on sites that have NEVER had a problem with it and no errors. I even looked around on Twitter thinking it was a Google Bot bug. Then it just went away. Out of curiosity, why would a 404 on a robots.txt be OK? Shouldn't you want "Fetch as Google" to return the proper HTTP/1.1 200 OK?
- Frozenlock 14y agoI don't have a robots.txt, so -for this url alone- I'm expecting a 404. What I've read in the google documents is that when a request to get the robots.txt returns a 404, it will consider it as OK to crawl.