3 ms·
Disallowing a link in robots.txt will not stop google from indexing the page. Google "reserve the right" to index the page if they see links going towards it to
by nulluk 14y ago
Disallowing a link in robots.txt will not stop google from indexing the page. Google "reserve the right" to index the page if they see links going towards it to stop webmasters shooting themselves in the foot, they however won't fetch the page in accordance to robots.txt.
The recommended way is to allow google to crawl the page but explicitly "noindex" the page via the robots meta tag (or even the x-robots header) - http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449 http://support.google.com/webmasters/bin/answer.py?hl=en&...
Edit: Matt, explaining in a video: http://www.youtube.com/watch?v=KBdEwpRQRD0 http://www.youtube.com/watch?v=KBdEwpRQRD0