3 ms·
Doesn't this passage suggest that Google ignores robots.txt in its own cross-comparisons of search relevancy: Google said in October that it found statistical
by gojomo 16y ago
Doesn't this passage suggest that Google ignores robots.txt in its own cross-comparisons of search relevancy:
Google said in October that it found statistical evidence that Bing suddenly became more Google-like. More listings in the first page of results of both search engines seemed to match, as did more of the number one results.
How would you get statistically significant results for such things, over time, without constant automated probe queries against Bing?
I think such probes are both legal and wise... but Google should drop the pretense that robots.txt is a sacred barrier across which no analysis can be done, no matter how indirect or for what purpose.
Also, I'd wager at some time in its history – if not constantly even today – Google has shown panels of users results from Google and its competitors in various combinations – side-by-side, with and without branding, intermixed randomly – and used their reactions to detect areas where the competitors are doing well, and Google could improve.
Further, either human eyes or algorithms then tried to determine adjustments to close any gaps in user satisfaction. The net effect of any such process is – surprise, surprise! – leveraging strengths of other engines to patch weaknesses in Google. This is normal, expected behavior by any serious search competitor.
- Jabbles 16y agoI think there is a major difference between identifying areas for improvement and copying a specific search result from a competitor.
- gojomo 16y agoI don't see it as that different. If the panel suggests a competitor's top-10 result is strongly preferred, and then Google examines why they didn't have that in their top-10, and tinkers with their weightings until they do, then they've used a competitor's output signals to train their own systems.
- moultano 16y ago>I don't see it as that different. It is very very different. You can't tinker with the weighting of something that doesn't exist, and if you don't have the necessary data, no amount of reweighting is going to improve things. Microsoft in this case no longer needs to come up with any data of their own, they can just use clicks on Google as a proxy for any combination of signals.
- gojomo 16y agoDoes Google use clicks from anywhere on the web other than their own sites as a ranking signal?
- moultano 16y agoI don't have an exhaustive knowledge of everything we do, and I couldn't tell you if I did. Amit has made it very clear however, that we will never do anything that would cause us to directly or indirectly copy a competitor's search results.
- gojomo 16y agoIt's clear to me Google wouldn't intentionally do something identical to what Bing's done. But given what Googlers can't say, and don't even know about what other groups within Google are doing, it's not 'very clear' to me that you aren't already doing very very analogous things, with regard to every other site on the internet. You've got the data; you're allowed to use it by your privacy policy; you've got the rationalizations handy. ("Sites didn't block us; fully-informed users opted-in; this is a crucial way to fight the manipulators; it's only helping us weight things we already found by other means; etc.") Amit's not made anything clear to me, with his finessed "put any results" wording. Danny Sullivan picked this up too, as he remarks in the headlined article: Google’s initial denial that it has never used toolbar data “to put any results on Google’s results pages” immediately took a blow given that site speed measurements done by the toolbar DO play a role in this. So what else might the toolbar do? There's wiggle room in the definitions of 'copy' and 'competitor' in your 'never' promise, too. Is it OK if Google Toolbar data hoovers up implied editorial-quality signals from user navigation on every site that isn't a 'competitor'? (And given Google's size, what site isn't a competitor in some respects for audience share?) Is it 'copying' if your use of clicktrails makes a preexisting result move from #11 to #9 after you observe it satisfying people in other browsing sessions? Move from #99 to #2? (Has the effect of any of Google's competitive analysis ever resulted in a single result moving closer to the position, higher or lower, that it had at a studied competitor? Some people could call that 'copying'.) Maybe none of the clickstream sources Google uses stick out as a dominating factor because Google has so effectively "commoditized its complements" – and no one other entity (except maybe Facebook) has access to as much clickstream data as Google does, simply from its own sites. Given that, it seems a little convenient that Google's standard is "every aggressively creative use of behavioral trails that led up to our 70%-90% share dominance was OK, but from now on let's be really rigorous about letting others observe our info-effluents."