4 ms·

With respect to the author, the conclusion here is very flawed. If you search for Bing in Google, you get Bing all over page 1. If you search for Google in Bin

With respect to the author, the conclusion here is very flawed. If you search for Bing in Google, you get Bing all over page 1. If you search for Google in Bing, you get Google all over page 1. That's not the result of Google capturing click stream data from Google Chrome and copying Bing's results, nor is it the result of Microsoft capturing click stream data from IE8 and copying Google's results. That's just the nature of indexing. As for robots.txt disallowing those URLs, there is no standard for robots.txt behavior. I have observed some user agents treat it as case insensitive, and others treat it as case sensitive. Honestly, this isn't even in the same ballpark as the Google accusations made earlier this week, and it smacks of just looking for things to accuse Google of in response to the "Binggate" (ugh, I typed it) drama. Can't we go back to more productive things?

deleted 16y ago
[deleted]
unp3rs0n 16y ago
I don't understand why everyone is using the term "copying the results". I think what Bing did was very smart, they incorporated user clickstream data. One could accuse this method of walking a thin line morally, but I suspect that Google's accusation wouldn't have stood any water as a lawsuit.
- luigi 16y ago
  Because by incorporating clickstream data from Google, they're effectively copying Google search results. Bing should blacklist Google from its clickstream data.
  - ajays 16y ago
    Not necessarily. It is possible that Google incorporates clickstream data too. The problem that Google's little experiment highlighted was: given the utter lack of any other signal, Bing uses the fact that that URL was ranked #1 by Google's search engine and clicked on by a user. Having said that: had I been doing this experiment at Google, I would have also added the following variations: - for some search terms, rank the honeypot URL #1 but don't click on it - for some search terms, rank the honeypot URL #1 on some _other_ search engine's list and click on it. How can they do that? There are search engines out there which use Google in the backend. Experiment #1 would have shown more blatant copying. Experiment #2 would have shown whether it's just Google, or any other search engine.
    - mukyu 16y ago
      Google has said they do not use clickstream data for ranking from Google toolbar. They did have variations of their tests. Cutts mentioned this during the bigthink panel. Sometimes they went to Bing first or not, sometimes they clicked on the links and sometimes not, and various other things.
      - gojomo 16y ago
        Google has said they do not use clickstream data for ranking from Google toolbar. Please provide a reference. I've been looking for this statement and haven't found it. When Googlers are directly asked, they pointedly don't answer or say they don't know. Amit Singhal's statement was carefully worded to be ambiguous on this matter, and Google has apparently confirmed that page-load-time data (at the very least) from the Toolbar does affect rankings. Such use by search is definitely allowed by Google's written privacy policy. The confirm dialog a user passes when installing the Toolbar refers to that privacy policy. It'd be very easy for an official Google spokesperson to say clearly that Toolbar data doesn't drive search rankings, if that were true. That they haven't strongly suggests it is used. Search expert Danny Sullivan made the same observation in his 'Bing: Why Google’s Wrong In Its Accusations' article: As For The Google Toolbar Meanwhile, I’m on my third day of waiting to hear back from Google about just what exactly it does with its own toolbar. Now that the company has fired off accusations against Bing about data collection, Google loses the right to stay as tight-lipped as it has been in the past about how the toolbar may be used in search results. Google’s initial denial that it has never used toolbar data “to put any results on Google’s results pages” immediately took a blow given that site speed measurements done by the toolbar DO play a role in this. So what else might the toolbar do? http://searchengineland.com/bing-why-googles-wrong-in-its-accusations-63279 http://searchengineland.com/bing-why-googles-wrong-in-its-ac...
        mukyu 16y ago
        You are being disingenuous. That you cannot find them explicitly denying something does not therefore make it true nor provide any evidence that it is true. “Absolutely not. The PageRank feature sends back URLs, but we’ve never used those URLs or data to put any results on Google’s results page. We do not do that, and we will not do that,” said Singhal. http://www.toprankblog.com/2006/04/matt-cutts-on-toolbar-data/ http://www.toprankblog.com/2006/04/matt-cutts-on-toolbar-dat... In this one, Matt Cutts all but explicitly says that they do not use it.
  - unp3rs0n 16y ago
    Let's say tomorrow DDG is the search engine with the largest market share. Then Bing would be getting all the clickstream data from DDG. I hope you do realize that this "algorithm" is not Google specific. Its just a novel ranking technique that incorporates a human user feedback loop and is a pretty well known technique in the information retrieval field.
    - moultano 16y ago
      It would be equally unethical to be copying DDG's results in this fashion.
      - seanos 16y ago
        Equally unethical sure (x=y), but differing opinions on whether that (x) is ethical or unethical. Personally, I see nothing wrong with it.
      - nostrademons 16y ago
        Highly ironic, though, as DDG uses Yahoo as a backend, which uses Bing, which uses Google, which would use...DDG? I think there's a cycle in that list somewhere...
- sajidnizami 16y ago
  What Bing did is very smart! It actually was, but not crediting Google there just makes it look like a cheap shot. You won't quote someone without citing their name now would you? Sadly you are right, they won't be able to push a lawsuit. Not enough grounds for it, however Bing should acknowledge what they did and are doing crediting Google. If Google hadn't caught them, we would all be thinking Bing did it on its own, which really isn't the case.
  - unp3rs0n 16y ago
    Taking liberty to concoct a scenario. If Walmart asked shoppers to take a photograph of the product layout on display at their favorite shop (which say happens to be Target because its the most popular in town) and used that to make small modifications to its own layout, would you say Walmart needs to credit Target, or that its copying Target? This is an arrangement between Walmart and its shoppers and there is nothing Target can do about it other than making a brouhaha. I don't see why things have to be different in the digital world. We all know how user interfaces historically have been blatantly ripped off.
  - jiji 16y ago
    Actually, Bing should credit the users providing all that clickstream data and all the websites they are using. It just happens that Google is one of many... what's your point?
- zmmmmm 16y ago
  It's a subtle point, but the whole point of this issue is that Bing actually does copy more than just "clickstream data". Think about what "clickstream" data is - user clicks a link, the link and the text of the link get sent to Microsoft. But in Google's text the search term was not in the link text, nor was it in the target page the user went to. So then how does the search term end up in Bing's index? They have to get it from somewhere - where? The logical conclusion is they get it from the URL of the page the click is performed on, or from the referrer header when the link is clicked (essentially these are the same thing). But how do you do that generically? The search URL contains the query terms in a format that is unique to Google. The only way Bing can be seeing it is if they deliberately are parsing out the Google search terms by specifically targeting how Google encodes them. So rather than just generically "incorporating clickstream data" this is actually using a special procedure to extract search terms from clicks that are determined to be google searches and then putting those search terms into the bing index.
riffraff 16y ago
It's possible the author may have intended that: .1 since search results are only accessed through a search action, and not links in the wild .2 since even if links in the wild are followed the google bot should have respected the disallow /search rule (which may have no standard, but google usually respect the format used in that robots.txt, and results are in both cases as of this writing) 3. therefore, they got to know such pages by using clickstream data But I'm just guessing.
anonymous246 16y ago
Hmm, I'm not sure you RTFA. He didn't search for the term "bing", he searched for the term "site:.bing.com/search". So it IS kinda a big deal that they're disrespecting the robots.txt and listing those pages anyway. I have also seen Google listing one of my domains for which I specifically disallowed all spiders (Bing doesn't show those domains FYI). My feeble attempt at separating my personal and professional personas was defeated by frigging Google's blatant disregard for Internet etiquette. Since I'm trying to be anon here, I won't be able to list the search term. Sorry.
- jedsmith 16y ago
  > So it IS kinda a big deal that they're disrespecting the robots.txt and listing those pages anyway. Did you RMFC? The robots.txt doesn't match what is indexed. As has been exhaustively pointed out in this thread by more than a few commenters, "Search" != "search". On that very search results page, we see #1 which is: > OLAC search - Bing OLAC is an unrelated site, and at one point it apparently linked to "www.bing.com/search" with the anchor text "OLAC search - Bing", which Google faithfully indexed but absolutely did not fetch, as you notice there is no description. What else would "OLAC search" mean? This gives us insight into how Google indexes and that they actually respect robots.txt case-sensitively. The second URL, m.bing.com/Search/Results.aspx, was indexed and fetched because "Search" != "search".
  - xilun0 16y ago
    > "Search" != "search" This is highly ridiculous. HTTP URLs are not mandated to be case-sensitive (though it's recommended), and clearly lots of sites use them in a case-insensitive way. Robots should either consider robots.txt in a case-insensitive way (even if I'm conscious that lot of them, including major ones, currently don't do that, which is precisely what I consider to be a problem, and this is supported by what happened here -- where google is risking to appear as a fool). The following article has perfectly good arguments in favor of case-insensitivity or even more clever handling: http://www.slicksurface.com/blog/2007-04/be-careful-robotstxt-is-case-sensitive http://www.slicksurface.com/blog/2007-04/be-careful-robotstx... Also; failing to respect conditions of use of a service because an automatic process is not safe enough is not a completely exonerating excuse for the operator of such process...
jfr 16y ago
Except for the schema and host parts (which are not part of robots.txt anyway) URLs are case sensitive (ref: RFC 3986, sections 6.2.2.2 and 6.2.3). The problem here is that Microsoft servers respond to /search, /Search and /SeaRCh without distinction. They are all distinct URLs. If it was the intended behavior (stupid, but understandable, coming from Microsoft), then robots.txt should contain all variants in capitalization for each path. A better solution would be to force a 301 redirect to a canonical path, and have this path in robots.txt. Google would work as expected. The original article is totally bogus. I can't imagine how it has over 90 votes.
- deleted 16y ago
  [deleted]

deleted 16y ago

[deleted]

unp3rs0n 16y ago

I don't understand why everyone is using the term "copying the results". I think what Bing did was very smart, they incorporated user clickstream data. One could accuse this method of walking a thin line morally, but I suspect that Google's accusation wouldn't have stood any water as a lawsuit.

luigi 16y ago
Because by incorporating clickstream data from Google, they're effectively copying Google search results. Bing should blacklist Google from its clickstream data.
- ajays 16y ago
  Not necessarily. It is possible that Google incorporates clickstream data too. The problem that Google's little experiment highlighted was: given the utter lack of any other signal, Bing uses the fact that that URL was ranked #1 by Google's search engine and clicked on by a user. Having said that: had I been doing this experiment at Google, I would have also added the following variations: - for some search terms, rank the honeypot URL #1 but don't click on it - for some search terms, rank the honeypot URL #1 on some _other_ search engine's list and click on it. How can they do that? There are search engines out there which use Google in the backend. Experiment #1 would have shown more blatant copying. Experiment #2 would have shown whether it's just Google, or any other search engine.
  - mukyu 16y ago
    Google has said they do not use clickstream data for ranking from Google toolbar. They did have variations of their tests. Cutts mentioned this during the bigthink panel. Sometimes they went to Bing first or not, sometimes they clicked on the links and sometimes not, and various other things.
    - gojomo 16y ago
      Google has said they do not use clickstream data for ranking from Google toolbar. Please provide a reference. I've been looking for this statement and haven't found it. When Googlers are directly asked, they pointedly don't answer or say they don't know. Amit Singhal's statement was carefully worded to be ambiguous on this matter, and Google has apparently confirmed that page-load-time data (at the very least) from the Toolbar does affect rankings. Such use by search is definitely allowed by Google's written privacy policy. The confirm dialog a user passes when installing the Toolbar refers to that privacy policy. It'd be very easy for an official Google spokesperson to say clearly that Toolbar data doesn't drive search rankings, if that were true. That they haven't strongly suggests it is used. Search expert Danny Sullivan made the same observation in his 'Bing: Why Google’s Wrong In Its Accusations' article: As For The Google Toolbar Meanwhile, I’m on my third day of waiting to hear back from Google about just what exactly it does with its own toolbar. Now that the company has fired off accusations against Bing about data collection, Google loses the right to stay as tight-lipped as it has been in the past about how the toolbar may be used in search results. Google’s initial denial that it has never used toolbar data “to put any results on Google’s results pages” immediately took a blow given that site speed measurements done by the toolbar DO play a role in this. So what else might the toolbar do? http://searchengineland.com/bing-why-googles-wrong-in-its-accusations-63279 http://searchengineland.com/bing-why-googles-wrong-in-its-ac...
      - mukyu 16y ago
        You are being disingenuous. That you cannot find them explicitly denying something does not therefore make it true nor provide any evidence that it is true. “Absolutely not. The PageRank feature sends back URLs, but we’ve never used those URLs or data to put any results on Google’s results page. We do not do that, and we will not do that,” said Singhal. http://www.toprankblog.com/2006/04/matt-cutts-on-toolbar-data/ http://www.toprankblog.com/2006/04/matt-cutts-on-toolbar-dat... In this one, Matt Cutts all but explicitly says that they do not use it.
- unp3rs0n 16y ago
  Let's say tomorrow DDG is the search engine with the largest market share. Then Bing would be getting all the clickstream data from DDG. I hope you do realize that this "algorithm" is not Google specific. Its just a novel ranking technique that incorporates a human user feedback loop and is a pretty well known technique in the information retrieval field.
  - moultano 16y ago
    It would be equally unethical to be copying DDG's results in this fashion.
    - seanos 16y ago
      Equally unethical sure (x=y), but differing opinions on whether that (x) is ethical or unethical. Personally, I see nothing wrong with it.
    - nostrademons 16y ago
      Highly ironic, though, as DDG uses Yahoo as a backend, which uses Bing, which uses Google, which would use...DDG? I think there's a cycle in that list somewhere...
sajidnizami 16y ago
What Bing did is very smart! It actually was, but not crediting Google there just makes it look like a cheap shot. You won't quote someone without citing their name now would you? Sadly you are right, they won't be able to push a lawsuit. Not enough grounds for it, however Bing should acknowledge what they did and are doing crediting Google. If Google hadn't caught them, we would all be thinking Bing did it on its own, which really isn't the case.
- unp3rs0n 16y ago
  Taking liberty to concoct a scenario. If Walmart asked shoppers to take a photograph of the product layout on display at their favorite shop (which say happens to be Target because its the most popular in town) and used that to make small modifications to its own layout, would you say Walmart needs to credit Target, or that its copying Target? This is an arrangement between Walmart and its shoppers and there is nothing Target can do about it other than making a brouhaha. I don't see why things have to be different in the digital world. We all know how user interfaces historically have been blatantly ripped off.
- jiji 16y ago
  Actually, Bing should credit the users providing all that clickstream data and all the websites they are using. It just happens that Google is one of many... what's your point?
zmmmmm 16y ago
It's a subtle point, but the whole point of this issue is that Bing actually does copy more than just "clickstream data". Think about what "clickstream" data is - user clicks a link, the link and the text of the link get sent to Microsoft. But in Google's text the search term was not in the link text, nor was it in the target page the user went to. So then how does the search term end up in Bing's index? They have to get it from somewhere - where? The logical conclusion is they get it from the URL of the page the click is performed on, or from the referrer header when the link is clicked (essentially these are the same thing). But how do you do that generically? The search URL contains the query terms in a format that is unique to Google. The only way Bing can be seeing it is if they deliberately are parsing out the Google search terms by specifically targeting how Google encodes them. So rather than just generically "incorporating clickstream data" this is actually using a special procedure to extract search terms from clicks that are determined to be google searches and then putting those search terms into the bing index.

riffraff 16y ago

It's possible the author may have intended that: .1 since search results are only accessed through a search action, and not links in the wild .2 since even if links in the wild are followed the google bot should have respected the disallow /search rule (which may have no standard, but google usually respect the format used in that robots.txt, and results are in both cases as of this writing) 3. therefore, they got to know such pages by using clickstream data But I'm just guessing.

anonymous246 16y ago

Hmm, I'm not sure you RTFA. He didn't search for the term "bing", he searched for the term "site:.bing.com/search". So it IS kinda a big deal that they're disrespecting the robots.txt and listing those pages anyway. I have also seen Google listing one of my domains for which I specifically disallowed all spiders (Bing doesn't show those domains FYI). My feeble attempt at separating my personal and professional personas was defeated by frigging Google's blatant disregard for Internet etiquette. Since I'm trying to be anon here, I won't be able to list the search term. Sorry.

jedsmith 16y ago
> So it IS kinda a big deal that they're disrespecting the robots.txt and listing those pages anyway. Did you RMFC? The robots.txt doesn't match what is indexed. As has been exhaustively pointed out in this thread by more than a few commenters, "Search" != "search". On that very search results page, we see #1 which is: > OLAC search - Bing OLAC is an unrelated site, and at one point it apparently linked to "www.bing.com/search" with the anchor text "OLAC search - Bing", which Google faithfully indexed but absolutely did not fetch, as you notice there is no description. What else would "OLAC search" mean? This gives us insight into how Google indexes and that they actually respect robots.txt case-sensitively. The second URL, m.bing.com/Search/Results.aspx, was indexed and fetched because "Search" != "search".
- xilun0 16y ago
  > "Search" != "search" This is highly ridiculous. HTTP URLs are not mandated to be case-sensitive (though it's recommended), and clearly lots of sites use them in a case-insensitive way. Robots should either consider robots.txt in a case-insensitive way (even if I'm conscious that lot of them, including major ones, currently don't do that, which is precisely what I consider to be a problem, and this is supported by what happened here -- where google is risking to appear as a fool). The following article has perfectly good arguments in favor of case-insensitivity or even more clever handling: http://www.slicksurface.com/blog/2007-04/be-careful-robotstxt-is-case-sensitive http://www.slicksurface.com/blog/2007-04/be-careful-robotstx... Also; failing to respect conditions of use of a service because an automatic process is not safe enough is not a completely exonerating excuse for the operator of such process...

jfr 16y ago

Except for the schema and host parts (which are not part of robots.txt anyway) URLs are case sensitive (ref: RFC 3986, sections 6.2.2.2 and 6.2.3). The problem here is that Microsoft servers respond to /search, /Search and /SeaRCh without distinction. They are all distinct URLs. If it was the intended behavior (stupid, but understandable, coming from Microsoft), then robots.txt should contain all variants in capitalization for each path. A better solution would be to force a 301 redirect to a canonical path, and have this path in robots.txt. Google would work as expected. The original article is totally bogus. I can't imagine how it has over 90 votes.

deleted 16y ago
[deleted]