Robots.txt is only an indication of which pages should/shouldn't be visited. Instructions about which pages can be included in a search engine's index (and hen

Robots.txt is only an indication of which pages should/shouldn't be visited. Instructions about which pages can be included in a search engine's index (and hence linked to from results pages) would go in a meta tag on the relevant page. Very large sites would love to see a 'noindex' parameter in robots.txt; sadly it's not there.