Although strictly speaking you're right, it breaks the intent of robots.txt as the users' clickstream is fed into the server-side indexing engine alongside robo

Although strictly speaking you're right, it breaks the intent of robots.txt as the users' clickstream is fed into the server-side indexing engine alongside robot crawled data. Arguing that according to the strict letter of the spec, robots.txt only has to be obeyed by true crawlers does hold water as strictly speaking Bing could disregard robots.txt altogether - it's not certainly not legally enforceable. The intent of robots.txt is clear and Bing should be trying to obey it wherever possible. In this case it appears the keyword was associated because it was typed into the Google search box, and from the there to the faked destination. IMHO when the clickstream was analysed it should have disregarded clickstreams that pass a robots.txt-excluded page as these could establish associations that were not supposed to revealed by crawlers. We aught to hold Bing and any search engine to the highest standard when talking about crawling etiquette. Also, see http://news.ycombinator.com/item?id=2169817 http://news.ycombinator.com/item?id=2169817