3 ms·
The whole point of crawling and indexing websites to begin with is to simulate what users can and do click on. The fact that it is now technologically possible
by InclinedPlane 16y ago
The whole point of crawling and indexing websites to begin with is to simulate what users can and do click on. The fact that it is now technologically possible to actually track users navigation of the web doesn't change the intent of the robots.txt convention in controlling and limiting the way that search engines collect data from a site.
Bing/MS can choose to dishonor that convention, either directly or indirectly as they are now with the Bing toolbar, but I think that's a mistake on their part. Google (and Bing.com for that matter) has used the most widely accepted mechanism, robots.txt, for ensuring that content it does not believe should be indexed is not. It abides by that convention with other sites and expects other search engines to abide by that convention with respect to google.com in return.
From here the following things can happen:
1. Microsoft agrees that robots.txt should govern clickstream data from the Bing toolbar. The world returns to sanity once again.
2. The search engine community (MS, Google, etc.) agree on a new standard similar to robots.txt specifically for governing the use of clickstream data, sites are updated with new directives allowing/disallowing such use and the world returns to sanity.
3. Microsoft specifically denies that they should be limited in using clickstream data from any source, through convention or through any means. Search companies and other sites are forced to fall back on other means to achieve the same results and things get messy (for example, google blocks all users who have the Bing toolbar installed, they prevent the Bing toolbar from being installed in Chrome, MS retaliates, etc.)
For the life of me I cannot think of a sane reason why MS would choose option number 3 other than that they are insane, horribly myopic, or just plain dumb.
- deleted 16y ago[deleted]
- SimonPStevens 16y agoI think it would be rather short sited of Google to start getting into wars over permissions for mining clickstream data from the source page, given the amount of data mining they do on Gmail links and search result links etc. I can imagine a reverse sting operation Bing could run: 1) Set up similar faked honeypot pages that only Bing knows about. 2) Send emails to Gmail accounts owned by the Bing management team. 3) Click on a few of those links. 4) Watch as Google mines the clicks and ends up with the links in their search index. 5) Sit back and accuse Google of reading private emails from Bing management to improve their search results. It's exactly the same thing, the data is owned by the user, and in both cases the user will have agreed to a terms and conditions that allows the companies to collect stats from that data.
- kd5bjo 16y agoI was always under the impression that robots.txt was intended not to control search results, but to help prevent overzealous crawlers from taking down your website. If you want to control what a search engine does with a page, you should use <meta name="ROBOTS"> and rel=nofollow, neither of which Google includes in its results pages. It would be reasonable for Bing to look for and honor these directives when collecting their clickstream data, and Google isn't availing itself of the option.
- InclinedPlane 16y agorel="nofollow" is only useful for pages which are intended to be indexed, google's search results pages are not, which is why they don't use it for the links in search results. The point is: up until now search indexing has had a robust opt-out mechanism. Bing has changed that and has muddied the waters with vague hand-waving. The question still remains: should there still be a way for sites to opt-out of search indexing? If so, then why can't robots.txt continue to be that mechanism? If robots.txt can't be that mechanism then what should be the new standard. If not, then that opens an entirely new can of worms. A can that I think we're better off not opening.
- beoba 16y agoIt's for controlling search results; blocking overzealous crawlers is a convenient side-effect. robots.txt also blocks indexing of non-html files.