3 ms·

Err, robots.txt doesn't even enter into the picture. It would matter only if Bing's server was requesting pages off Google's site. That is not happening. The on

Err, robots.txt doesn't even enter into the picture. It would matter only if Bing's server was requesting pages off Google's site. That is not happening. The only things that are going are the Google URL and only the clicked URL(not the other Google results that the user didn't click on). The user gave permission to Microsoft to use this info by installing the toolbar. I don't see why or how the Bing tool bar should visit Google.com/robots.txt to see the blocked folders unless they were crawling Google pages.

zzleeper 16y ago
But it's the perfect way to circumvent robots.txt! With enough users the law of large numbers applies (and how many hundreds of millions of users are using IE?) so you can a.s. copy Google's index. In particular, you copy the most popular results, so if you weight the % of google's index that you copied, then it's going to be a huge number, and almost for free. It's evil. But devilishly smart.
InclinedPlane 16y ago
The contention you're making, and apparently Microsoft as well, is that the Bing toolbar is not a search indexer and does not need to obey the conventions of robots.txt? Personally I don't think that flies. Should I be able to create a toolbar that rips the data from hulu.com and hosts it elsewhere?
- catshirt 16y ago
  this is not my area of expertise but i really don't think that's a proper analogy. a more relevant phrasing would be, "should i be able to create a toolbar that inspects (previously/explicitly) requested documents from hulu for external links". and i would be inclined to say, "yes"...