4 ms·
There is no robot involved. An actual human instructs their user agent to retrieve a document on Google. A piece of code, running in that user agent, does som
by patio11 16y ago
There is no robot involved. An actual human instructs their user agent to retrieve a document on Google. A piece of code, running in that user agent, does something. If anyone has a complaint here, it is the user, not Google. (Google really really doesn't want to spin this as "MS is spying on our users" because as soon as they say that some person is going to point out "Google is spying on the entire world" and that person will be right.)
- mechanical_fish 16y agoThank you - I've been looking for the metaphor to express my impression of this whole matter, and you've given it to me. It's Spy vs Spy: http://en.wikipedia.org/wiki/Spy_vs._Spy http://en.wikipedia.org/wiki/Spy_vs._Spy ...a pitched battle between the opposite-but-indistinguishable agents of two superpowers, on a plane so far removed from the realm of actual people and their concerns that the customers don't even appear in the frame.
- motti 16y agoAlthough strictly speaking you're right, it breaks the intent of robots.txt as the users' clickstream is fed into the server-side indexing engine alongside robot crawled data. Arguing that according to the strict letter of the spec, robots.txt only has to be obeyed by true crawlers does hold water as strictly speaking Bing could disregard robots.txt altogether - it's not certainly not legally enforceable. The intent of robots.txt is clear and Bing should be trying to obey it wherever possible. In this case it appears the keyword was associated because it was typed into the Google search box, and from the there to the faked destination. IMHO when the clickstream was analysed it should have disregarded clickstreams that pass a robots.txt-excluded page as these could establish associations that were not supposed to revealed by crawlers. We aught to hold Bing and any search engine to the highest standard when talking about crawling etiquette. Also, see http://news.ycombinator.com/item?id=2169817 http://news.ycombinator.com/item?id=2169817
- Dylan16807 16y agoIf I'm google and I block /search I want no company to collect the data. If I'm a site that blocks /dynamic because I want to make sure my stats are accurate, not factoring bots, and I want to keep the load down on slow-generating pages, then I'm perfectly happy with the data being collected. Is it the intent of robots.txt itself to block the clickstream data, or is it just google's intent?
- haberman 16y agoBy that logic IE could send the entire page of search results back to Microsoft and it wouldn't be a violation of robots.txt. Because hey, it was a user-agent doing it, not a robot! If this is so in the clear, why not have Microsoft do that also?