3 ms·
I was always under the impression that robots.txt was intended not to control search results, but to help prevent overzealous crawlers from taking down your web
by kd5bjo 16y ago
I was always under the impression that robots.txt was intended not to control search results, but to help prevent overzealous crawlers from taking down your website.
If you want to control what a search engine does with a page, you should use <meta name="ROBOTS"> and rel=nofollow, neither of which Google includes in its results pages. It would be reasonable for Bing to look for and honor these directives when collecting their clickstream data, and Google isn't availing itself of the option.
- InclinedPlane 16y agorel="nofollow" is only useful for pages which are intended to be indexed, google's search results pages are not, which is why they don't use it for the links in search results. The point is: up until now search indexing has had a robust opt-out mechanism. Bing has changed that and has muddied the waters with vague hand-waving. The question still remains: should there still be a way for sites to opt-out of search indexing? If so, then why can't robots.txt continue to be that mechanism? If robots.txt can't be that mechanism then what should be the new standard. If not, then that opens an entirely new can of worms. A can that I think we're better off not opening.
- beoba 16y agoIt's for controlling search results; blocking overzealous crawlers is a convenient side-effect. robots.txt also blocks indexing of non-html files.