5 ms·

http://www.google.com/robots.txt http://www.google.com/robots.txt First two lines: User-agent: * Disallow: /search

by Joakal 16y ago

user24 16y ago
except that that's not what robots.txt is for. Otherwise your adblock extension could be excluded via robots.txt edit: bing toolbar is effectively an extension for MSIE. robots.txt is designed to stop spiders. Spiders aren't extensions. Therefore that's not what robots.txt is for. What did I say wrong to get downvoted?
- patrickas 16y ago
  >Otherwise your adblock extension could be excluded via robots.txt I think that's what most people though was wrong in your comment. Care to elaborate on how an adblock extension could be excluded via robots.txt in that fictive scenario?
  - user24 16y ago
    If extensions checked to see if they had permission to alter/interfere/access webpages by looking it up in robots.txt, which is what is being suggested by people who say "Bing toolbar violated robots.txt", then that same mechanism could be used to specify: User-Agent:adblock Disallow: / Or even, if the "User-Agent: (star)" is supposed to tell Bing Toolbar not to play about with the page, then why doesn't that also apply to all your other extensions/addon/BHO's/plugins whatever you want to call them?
    - lukeschlather 16y ago
      It should, if they're sending data back to the mothership. Alexa's toolbar among others should respect robots.txt
patio11 16y ago
There is no robot involved. An actual human instructs their user agent to retrieve a document on Google. A piece of code, running in that user agent, does something. If anyone has a complaint here, it is the user, not Google. (Google really really doesn't want to spin this as "MS is spying on our users" because as soon as they say that some person is going to point out "Google is spying on the entire world" and that person will be right.)
- mechanical_fish 16y ago
  Thank you - I've been looking for the metaphor to express my impression of this whole matter, and you've given it to me. It's Spy vs Spy: http://en.wikipedia.org/wiki/Spy_vs._Spy http://en.wikipedia.org/wiki/Spy_vs._Spy ...a pitched battle between the opposite-but-indistinguishable agents of two superpowers, on a plane so far removed from the realm of actual people and their concerns that the customers don't even appear in the frame.
- motti 16y ago
  Although strictly speaking you're right, it breaks the intent of robots.txt as the users' clickstream is fed into the server-side indexing engine alongside robot crawled data. Arguing that according to the strict letter of the spec, robots.txt only has to be obeyed by true crawlers does hold water as strictly speaking Bing could disregard robots.txt altogether - it's not certainly not legally enforceable. The intent of robots.txt is clear and Bing should be trying to obey it wherever possible. In this case it appears the keyword was associated because it was typed into the Google search box, and from the there to the faked destination. IMHO when the clickstream was analysed it should have disregarded clickstreams that pass a robots.txt-excluded page as these could establish associations that were not supposed to revealed by crawlers. We aught to hold Bing and any search engine to the highest standard when talking about crawling etiquette. Also, see http://news.ycombinator.com/item?id=2169817 http://news.ycombinator.com/item?id=2169817
  - Dylan16807 16y ago
    If I'm google and I block /search I want no company to collect the data. If I'm a site that blocks /dynamic because I want to make sure my stats are accurate, not factoring bots, and I want to keep the load down on slow-generating pages, then I'm perfectly happy with the data being collected. Is it the intent of robots.txt itself to block the clickstream data, or is it just google's intent?

http://www.google.com/robots.txt http://www.google.com/robots.txt First two lines: User-agent: * Disallow: /search

by Joakal 16y ago

http://www.google.com/robots.txt http://www.google.com/robots.txt First two lines: User-agent: * Disallow: /search

except that that's not what robots.txt is for. Otherwise your adblock extension could be excluded via robots.txt edit: bing toolbar is effectively an extension for MSIE. robots.txt is designed to stop spiders. Spiders aren't extensions. Therefore that's not what robots.txt is for. What did I say wrong to get downvoted?

patrickas 16y ago
>Otherwise your adblock extension could be excluded via robots.txt I think that's what most people though was wrong in your comment. Care to elaborate on how an adblock extension could be excluded via robots.txt in that fictive scenario?
- user24 16y ago
  If extensions checked to see if they had permission to alter/interfere/access webpages by looking it up in robots.txt, which is what is being suggested by people who say "Bing toolbar violated robots.txt", then that same mechanism could be used to specify: User-Agent:adblock Disallow: / Or even, if the "User-Agent: (star)" is supposed to tell Bing Toolbar not to play about with the page, then why doesn't that also apply to all your other extensions/addon/BHO's/plugins whatever you want to call them?
  - lukeschlather 16y ago
    It should, if they're sending data back to the mothership. Alexa's toolbar among others should respect robots.txt

patio11 16y ago

There is no robot involved. An actual human instructs their user agent to retrieve a document on Google. A piece of code, running in that user agent, does something. If anyone has a complaint here, it is the user, not Google. (Google really really doesn't want to spin this as "MS is spying on our users" because as soon as they say that some person is going to point out "Google is spying on the entire world" and that person will be right.)

mechanical_fish 16y ago
Thank you - I've been looking for the metaphor to express my impression of this whole matter, and you've given it to me. It's Spy vs Spy: http://en.wikipedia.org/wiki/Spy_vs._Spy http://en.wikipedia.org/wiki/Spy_vs._Spy ...a pitched battle between the opposite-but-indistinguishable agents of two superpowers, on a plane so far removed from the realm of actual people and their concerns that the customers don't even appear in the frame.
motti 16y ago
Although strictly speaking you're right, it breaks the intent of robots.txt as the users' clickstream is fed into the server-side indexing engine alongside robot crawled data. Arguing that according to the strict letter of the spec, robots.txt only has to be obeyed by true crawlers does hold water as strictly speaking Bing could disregard robots.txt altogether - it's not certainly not legally enforceable. The intent of robots.txt is clear and Bing should be trying to obey it wherever possible. In this case it appears the keyword was associated because it was typed into the Google search box, and from the there to the faked destination. IMHO when the clickstream was analysed it should have disregarded clickstreams that pass a robots.txt-excluded page as these could establish associations that were not supposed to revealed by crawlers. We aught to hold Bing and any search engine to the highest standard when talking about crawling etiquette. Also, see http://news.ycombinator.com/item?id=2169817 http://news.ycombinator.com/item?id=2169817
- Dylan16807 16y ago
  If I'm google and I block /search I want no company to collect the data. If I'm a site that blocks /dynamic because I want to make sure my stats are accurate, not factoring bots, and I want to keep the load down on slow-generating pages, then I'm perfectly happy with the data being collected. Is it the intent of robots.txt itself to block the clickstream data, or is it just google's intent?