3 ms·
How do you know it does not? I'm assuming robots.txt is about preventing the page contents from being crawled and added to the index. If all they use the click
by pedrocr 16y ago
How do you know it does not? I'm assuming robots.txt is about preventing the page contents from being crawled and added to the index. If all they use the click info for is to associate referrers (google URLs in this case) to pages in the index and they don't crawl the google search itself I don't see how that breaks the robots.txt contract.
- las3rjock 16y agoThe page contents are being crawled and added to the index, but by Bing Toolbar users, not a computer program. I consider that to be an underhanded way to circumvent robots.txt, but others might not.
- pedrocr 16y agoWhat makes you say that? We haven't seen anything to indicate Google search pages are in the Bing index.