6 ms·

But the effect is the same as intentional scraping and outright stealing: search results that could have only come from Google are appearing as Bing results. B

by luigi 16y ago

But the effect is the same as intentional scraping and outright stealing: search results that could have only come from Google are appearing as Bing results. Bing needs to blacklist Google from its clickstream. Simple.

checker 16y ago
Would Bing then get called out by Google for the inevitable 'anticompetitive' lowering of Google's Bing search rankings?
newt 16y ago
the effect is the same as intentional scraping and outright stealing The google engineers intentionally sent this click data to Bing, so is Bing really stealing? It's odd to act surprised when Bing uses the data that was intentionally sent to it. Bing could specifically ignore Google search results pages when it is tracking clicks, but is that legitimate? Google scrapes everything, why shouldn't Bing?
- ohashi 16y ago
  They were testing to see if it was true. Many users are doing it, bing is taking data for non-google engineers too.
  - newt 16y ago
    The point of collecting click data is not to target google engineers, it is to collect data from masses of people doing regular searches and to improve them by seeing which links get clicked on, so obviously Bing is "taking data for non-google engineers". Furthermore, there's no indication that google search results pages are even distinguished from other pages in this. In fact, the data that it takes from Google engineers for carefully engineered corner-case searches is the exception.
nlogn 16y ago
I don't even think they should just blacklist Google. They should just respect robots.txt. edit: I should have clarified. I know that the Bing crawler likely respects robots.txt, but if they are using clickstream info to build their index, it seems right that they should respect robots.txt there as well, no?
- cgoddard 16y ago
  You could strongly argue that collecting clickstream and other user browser session info via a toolbar is not a form of web robot (crawler, spider, etc.), and thus robots.txt does not apply.
- stanleydrew 16y ago
  I'm pretty sure the Bing Crawler does respect robots.txt. The data Bing collected didn't come from spidering Google.
- FreebytesSector 16y ago
  I agree with your comments that toolbars should respect the robots.txt because even if a human is doing the crawling, it is still an automated system that is indexing information from that site. I would not want toolbars attempting to send data back to Bing based on my queries on a company Intranet or a site that would normally not be indexed. Personal data entered into what the toolbar thought was a query field could be sent onward as well even if the robots.txt on the site restricted it. I think they should respect robots.txt in this case even if they are only monitoring user behavior.
gojomo 16y ago
...the effect is the same as intentional scraping and outright stealing: search results that could have only come from Google are appearing as Bing results... That's one effect. While it's vivid, it might be a tiny side-effect only notable in contrived cases. Overall, this kind of URL-after-URL signal, extracted from every participating user, and every trail through both search sites and non-search sites, might be discovering valuable terms-in-preceding-URL-to-later-visited-URL associations. These associations might result in many search improvements, other than the one-for-one result porting Google's experiment has found. We don't really know the relative magnitude of porting-results versus other-benefits, yet I think that's important to the analysis. If a useful automated or user-driven process generates a little indirect infringement around the edges, is that enough to demand the process be stopped entirely? Note, that's not the standard Google wants applied to user uploads to YouTube, or excerpting of news and websites onto Google services. Google says: "defend yourself, by adding opt-outs (robots.txt) or sending takedown notices, and we'll undo the incidental infringements eventually".

by luigi 16y ago

checker 16y ago

Would Bing then get called out by Google for the inevitable 'anticompetitive' lowering of Google's Bing search rankings?

newt 16y ago

the effect is the same as intentional scraping and outright stealing The google engineers intentionally sent this click data to Bing, so is Bing really stealing? It's odd to act surprised when Bing uses the data that was intentionally sent to it. Bing could specifically ignore Google search results pages when it is tracking clicks, but is that legitimate? Google scrapes everything, why shouldn't Bing?

ohashi 16y ago
They were testing to see if it was true. Many users are doing it, bing is taking data for non-google engineers too.
- newt 16y ago
  The point of collecting click data is not to target google engineers, it is to collect data from masses of people doing regular searches and to improve them by seeing which links get clicked on, so obviously Bing is "taking data for non-google engineers". Furthermore, there's no indication that google search results pages are even distinguished from other pages in this. In fact, the data that it takes from Google engineers for carefully engineered corner-case searches is the exception.

nlogn 16y ago

I don't even think they should just blacklist Google. They should just respect robots.txt. edit: I should have clarified. I know that the Bing crawler likely respects robots.txt, but if they are using clickstream info to build their index, it seems right that they should respect robots.txt there as well, no?

cgoddard 16y ago
You could strongly argue that collecting clickstream and other user browser session info via a toolbar is not a form of web robot (crawler, spider, etc.), and thus robots.txt does not apply.
stanleydrew 16y ago
I'm pretty sure the Bing Crawler does respect robots.txt. The data Bing collected didn't come from spidering Google.
FreebytesSector 16y ago
I agree with your comments that toolbars should respect the robots.txt because even if a human is doing the crawling, it is still an automated system that is indexing information from that site. I would not want toolbars attempting to send data back to Bing based on my queries on a company Intranet or a site that would normally not be indexed. Personal data entered into what the toolbar thought was a query field could be sent onward as well even if the robots.txt on the site restricted it. I think they should respect robots.txt in this case even if they are only monitoring user behavior.

gojomo 16y ago

...the effect is the same as intentional scraping and outright stealing: search results that could have only come from Google are appearing as Bing results... That's one effect. While it's vivid, it might be a tiny side-effect only notable in contrived cases. Overall, this kind of URL-after-URL signal, extracted from every participating user, and every trail through both search sites and non-search sites, might be discovering valuable terms-in-preceding-URL-to-later-visited-URL associations. These associations might result in many search improvements, other than the one-for-one result porting Google's experiment has found. We don't really know the relative magnitude of porting-results versus other-benefits, yet I think that's important to the analysis. If a useful automated or user-driven process generates a little indirect infringement around the edges, is that enough to demand the process be stopped entirely? Note, that's not the standard Google wants applied to user uploads to YouTube, or excerpting of news and websites onto Google services. Google says: "defend yourself, by adding opt-outs (robots.txt) or sending takedown notices, and we'll undo the incidental infringements eventually".