3 ms·
This looks great and I gave it a try. Is there a reason you ignore robots.txt? Example result is 'thin content' on our shopping cart, which we say not to index
by hnick 7y ago
This looks great and I gave it a try. Is there a reason you ignore robots.txt? Example result is 'thin content' on our shopping cart, which we say not to index obviously :)
- seanwilson 7y agoCheckbot ignores robots.txt here because Google can still index pages even when they're included in robots.txt files. If you don't want pages to be indexed, you should use a "noindex" tag instead of robots.txt. It's a common misconception that robots.txt stops indexing when it doesn't. See: https://support.google.com/webmasters/answer/93710?hl=en https://support.google.com/webmasters/answer/93710?hl=en > Important! For the noindex directive to be effective, the page must not be blocked by a robots.txt file. If the page is blocked by a robots.txt file, the crawler will never see the noindex directive, and the page can still appear in search results, for example if other pages link to it.
- hnick 7y agoThanks, that makes perfect sense. And I'm definitely guilty of that misconception. This would definitely be last on a long list of things to clean up for my site.