4 ms·

Anyone can test that Microsoft's software sends the clicks back to Microsoft, although I believe Microsoft sends the data back by SSL, so it's harder to verify

Anyone can test that Microsoft's software sends the clicks back to Microsoft, although I believe Microsoft sends the data back by SSL, so it's harder to verify even that than you'd expect. Google's search results are blocked in robots.txt, so I don't believe Bing has been able to crawl our search results directly. All the evidence points to users' clicks on Google, which are then sent to Microsoft. Microsoft has (so far) declined to admit whether our allegation is true. Getting them to talk about exactly what they do and what software they use or don't use would be the easiest way. I'd like them to confirm or deny, which is why I wanted to go to this search panel later today and ask them.

poacher69 16y ago
are you claiming that google never scrapes bing search results pages? or any other search result pages?
- illdave 16y ago
  From my experience, Googlebot doesn't crawl pages that are blocked in robots.txt files. Check out Bing's robots.txt: http://bing.com/robots.txt http://bing.com/robots.txt - notice how /search is disallowed. That typically means that Googlebot isn't able to access that page. The same for the other search engines, it's more down to if they specify (through robots.txt) that Googlebot isn't allowed to crawl those results.
- Matt_Cutts 16y ago
  poacher69, we crawl the public web. Anyone that blocks us out with robots.txt, we won't crawl. If you check bing.com/robots.txt, it has "Disallow: /search" . So no, we won't crawl Bing's search results pages. If anything, users tend to complain when search results from Lycos or wherever show up in Google.
  - underdown 16y ago
    http://www.bing.com/robots.txt http://www.bing.com/robots.txt User-agent: * Disallow: /search Funny thing: http://www.google.com/search?q=site%3Abing.com%2Fsearch%2F http://www.google.com/search?q=site%3Abing.com%2Fsearch%2F I was gonna call out Matt for crawling bing's search results but I'm guessing Microsoft hasn't realized they return results from the /Search/ folder. ;)
    - nitrogen 16y ago
      Once again Microsoft is bitten by expecting case insensitivity.
  - poacher69 16y ago
    matt, how does google do competitive relevance evaluations without scraping Bing?
kgrin 16y ago
> so I don't believe Bing has been able to crawl our search results directly Isn't compliance with robots.txt more of a voluntary thing? I'm not accusing MS of ignoring it when convenient, but if you/we/someone is accusing them of acting unethically wrt search results in the first place, telling the crawler to ignore robots.txt wouldn't be that far away, would it? (And likewise faking the user-agent, etc.) For better or for worse, UA identification, robots.txt compliance - all those things are voluntary. I'm not suggesting they shouldn't be, but it certainly makes a difference in terms of whether something's possible or not. (And, if you ask me, places an even higher obligation on the actors to behave ethically, lest trust completely evaporates and the whole thing goes to hell in a handbasket).
- jasonwatkinspdx 16y ago
  I am not a lawyer, but as I understand it there is some precedent in the US of intentionally ignoring robots.txt being unauthorized computer access, exposing you to all the liability that entails (possibly criminal).
  - c2 16y ago
    I'd like to see an actual case reference for this. I've never heard of ignoring robots.txt resulting in any kind of legal action. It would take a pretty big leap to go from robots.txt is advisory to ignoring it constitutes a criminal action.
    - jasonwatkinspdx 16y ago
      Internet Archive was sued unsuccessfully. As I understand it a lawsuit is still in process against Google on the topic. So I guess the precedent is weaker than I thought, but still: tread carefully.
deleted 16y ago
[deleted]
brudgers 16y ago
Matt, don't Google Toolbar and the Chrome Browser similarly send information to Google for use in improving their services?
- stanleydrew 16y ago
  If you read the article and other comments here it's been made perfectly clear that the Google toolbar and Chrome browser are not sending similar data back to Google.
  - brudgers 16y ago
    I did not say "similar data" because "similar" is a bit too slippery a word in a technical context. There's too much plausible deniablity. What I am asking is if Google's tools send data back to Googleplex to be mined for the sake of search engine improvements.
    - stanleydrew 16y ago
      Then what use is the word "similarly" in your comment? Similarly send? As in via HTTP requests? I think that's either obvious or irrelevant or both. Again, if you actually read the article, you will come across the section titled "What About The Google Toolbar & Chrome?" I encourage you to read it. [edit] Also, see this comment and patio11's subcomment further down the page, both of which were written an hour before yours: http://news.ycombinator.com/item?id=2165469#score_2165578 http://news.ycombinator.com/item?id=2165469#score_2165578.
      - pbz 16y ago
        Quote from the article: "In fact, Google stressed that the only information that flows back at all from Chrome is what people are searching for from within the browser, if they are using Google as their search engine." I'm pretty positive that's not true. If you run Fiddler when browsing with Chrome you will see constant hits to toolbarqueries.clients.google.com whether you're using Google or not. I could be browsing some MS site and toolbarqueries.clients.google.com gets hit. Chromium doesn't do this. Edit: You can uncheck everything under privacy and it will still send those requests. Edit2: What it sends back looks something like this: <?xml version="1.0" encoding="UTF-8"?><autofillquery clientversion="6.1.1715.1442/en (GGLL)"><form signature="8551191143090325242"><field signature="620769395"/><field signature="2995202485"/><field signature="2175865763"/><field signature="904516291"/><field signature="2953051246"/><field signature="2649047790"/><field signature="2308153337"/><field signature="1003471793"/><field signature="3255484099"/><field signature="1305698505"/><field signature="3676143819"/><field signature="1275502930"/></form></autofillquery> Looks like auto-fill data, but this happens when I click around a site, NOT when searching Google or typing something in the address bar. For some sites (interestingly, not all) it sends 3 requests for each page load.
DenisM 16y ago
>I believe Microsoft sends the data back by SSL, so it's harder to verify even that than you'd expect. Please. Adding my own SSL cert to my own laptop is not harder than I'd expect. Certainly not harder than many other things you did in setting up this experiment.

poacher69 16y ago

are you claiming that google never scrapes bing search results pages? or any other search result pages?

illdave 16y ago
From my experience, Googlebot doesn't crawl pages that are blocked in robots.txt files. Check out Bing's robots.txt: http://bing.com/robots.txt http://bing.com/robots.txt - notice how /search is disallowed. That typically means that Googlebot isn't able to access that page. The same for the other search engines, it's more down to if they specify (through robots.txt) that Googlebot isn't allowed to crawl those results.
Matt_Cutts 16y ago
poacher69, we crawl the public web. Anyone that blocks us out with robots.txt, we won't crawl. If you check bing.com/robots.txt, it has "Disallow: /search" . So no, we won't crawl Bing's search results pages. If anything, users tend to complain when search results from Lycos or wherever show up in Google.
- underdown 16y ago
  http://www.bing.com/robots.txt http://www.bing.com/robots.txt User-agent: * Disallow: /search Funny thing: http://www.google.com/search?q=site%3Abing.com%2Fsearch%2F http://www.google.com/search?q=site%3Abing.com%2Fsearch%2F I was gonna call out Matt for crawling bing's search results but I'm guessing Microsoft hasn't realized they return results from the /Search/ folder. ;)
  - nitrogen 16y ago
    Once again Microsoft is bitten by expecting case insensitivity.
- poacher69 16y ago
  matt, how does google do competitive relevance evaluations without scraping Bing?

kgrin 16y ago

> so I don't believe Bing has been able to crawl our search results directly Isn't compliance with robots.txt more of a voluntary thing? I'm not accusing MS of ignoring it when convenient, but if you/we/someone is accusing them of acting unethically wrt search results in the first place, telling the crawler to ignore robots.txt wouldn't be that far away, would it? (And likewise faking the user-agent, etc.) For better or for worse, UA identification, robots.txt compliance - all those things are voluntary. I'm not suggesting they shouldn't be, but it certainly makes a difference in terms of whether something's possible or not. (And, if you ask me, places an even higher obligation on the actors to behave ethically, lest trust completely evaporates and the whole thing goes to hell in a handbasket).

jasonwatkinspdx 16y ago
I am not a lawyer, but as I understand it there is some precedent in the US of intentionally ignoring robots.txt being unauthorized computer access, exposing you to all the liability that entails (possibly criminal).
- c2 16y ago
  I'd like to see an actual case reference for this. I've never heard of ignoring robots.txt resulting in any kind of legal action. It would take a pretty big leap to go from robots.txt is advisory to ignoring it constitutes a criminal action.
  - jasonwatkinspdx 16y ago
    Internet Archive was sued unsuccessfully. As I understand it a lawsuit is still in process against Google on the topic. So I guess the precedent is weaker than I thought, but still: tread carefully.

deleted 16y ago

[deleted]

brudgers 16y ago

Matt, don't Google Toolbar and the Chrome Browser similarly send information to Google for use in improving their services?

stanleydrew 16y ago
If you read the article and other comments here it's been made perfectly clear that the Google toolbar and Chrome browser are not sending similar data back to Google.
- brudgers 16y ago
  I did not say "similar data" because "similar" is a bit too slippery a word in a technical context. There's too much plausible deniablity. What I am asking is if Google's tools send data back to Googleplex to be mined for the sake of search engine improvements.
  - stanleydrew 16y ago
    Then what use is the word "similarly" in your comment? Similarly send? As in via HTTP requests? I think that's either obvious or irrelevant or both. Again, if you actually read the article, you will come across the section titled "What About The Google Toolbar & Chrome?" I encourage you to read it. [edit] Also, see this comment and patio11's subcomment further down the page, both of which were written an hour before yours: http://news.ycombinator.com/item?id=2165469#score_2165578 http://news.ycombinator.com/item?id=2165469#score_2165578.
    - pbz 16y ago
      Quote from the article: "In fact, Google stressed that the only information that flows back at all from Chrome is what people are searching for from within the browser, if they are using Google as their search engine." I'm pretty positive that's not true. If you run Fiddler when browsing with Chrome you will see constant hits to toolbarqueries.clients.google.com whether you're using Google or not. I could be browsing some MS site and toolbarqueries.clients.google.com gets hit. Chromium doesn't do this. Edit: You can uncheck everything under privacy and it will still send those requests. Edit2: What it sends back looks something like this: <?xml version="1.0" encoding="UTF-8"?><autofillquery clientversion="6.1.1715.1442/en (GGLL)"><form signature="8551191143090325242"><field signature="620769395"/><field signature="2995202485"/><field signature="2175865763"/><field signature="904516291"/><field signature="2953051246"/><field signature="2649047790"/><field signature="2308153337"/><field signature="1003471793"/><field signature="3255484099"/><field signature="1305698505"/><field signature="3676143819"/><field signature="1275502930"/></form></autofillquery> Looks like auto-fill data, but this happens when I click around a site, NOT when searching Google or typing something in the address bar. For some sites (interestingly, not all) it sends 3 requests for each page load.

DenisM 16y ago

>I believe Microsoft sends the data back by SSL, so it's harder to verify even that than you'd expect. Please. Adding my own SSL cert to my own laptop is not harder than I'd expect. Certainly not harder than many other things you did in setting up this experiment.