4 ms·

That clarification is appreciated, but it seems to contradict or ignore several of the results listed? In particular, DuckDuckGo attempted to check for the eff

That clarification is appreciated, but it seems to contradict or ignore several of the results listed? In particular, DuckDuckGo attempted to check for the effects of fresh browser windows, changes from localization, and A/B testing or incomplete rollouts. - The study results found that the 'distance' between two people's incognito results was 2.8 times larger than the distance between one person's normal and incognito results. If this is correct, the tweet's claim that you can use Incognito to see the impact of personalization for yourself is seriously misleading. - The study attempted to control for localization by tokenizing all local results, so that two result sets which each had a local story in the same spot would be treated as identical. - The study attempted to control for rollout and testing effects by running all searches at the same time and assuming that those forces would lead to most users seeing similar results, with a few differing. Instead, they found substantial variance across all users. I can certainly come up with explanations for how each of these changes could be non-personalized. Incognito variance could be a consequence of region and device effects which are user-independent. Tests might be scaled up near 50/50, and rollouts may not hit all datacenters at once, invalidating the "changes for a few users" assumption. And most significantly, DuckDuckGo's localization control looks completely inadequate to me. It would completely fail the given "football" example, and might fail for the given search terms: 'vaccine' produced highly-localized results they adjusted for, but 'gun control' could produce pseudo-local results like a preference for national stories with the user's state as a keyword, and that wouldn't have been identified. A better version of this study would presumably try to alter its variables separately, for instance by using multiple devices in one location and one device in multiple locations via VPN. As is, the controls seem seriously lacking. But I'm just making up reasons, and frankly they don't seem likely to explain the sheer size of the differences. When I run those searches, I don't get "football" and "Paris" level customization, I get a bunch of national-scope results that have no obvious reason to vary between incognito users. I wish the tweets here had touched on any of that. As is, they explain general search mechanics while frustratingly bypassing the most significant claims.

ryacko 8y ago
You just have to find out the one unique website a particular user always goes to and you can identify that the person is using a computer. Or it is 20 bits of entropy when you go to a website only a thousand other people in the world visit.
saalweachter 8y ago
Frankly, everyone always misunderstands and underestimates the impact of location. Location affects all queries, not just "local" queries. That's literally what is being demonstrated here.
- Bartweiss 8y ago
  Yes, and I responded quite directly to that. DuckDuckGo's control definitely wasn't adequate, but in return Google's examples ('football' and 'Paris') have unambiguous localization value that's not present for two different Americans searching "immigration". Quite a lot of the variance was link reordering and result changes that didn't have any clear regional aspects, whether at the domain or story level. If localization is the active factor, it's still pretty interesting to know that where I live determines whether to show a Wikipedia link and how to order HuffPo against the Tribune.
puzzle 8y ago
In comparing two people's incognito results, you have no idea if they hit the same search clusters. Were they in the same room? Same ISP? Same geographical region? Same continent? If I issue the same search back to back in different windows, I have a good chance of hitting the same cluster. But I still have no guarantee. Many years ago, SEO people figured that some Google IPs would serve new versions of the ranking algorithms earlier than the rest. Even those were not stable and definitely not a guarantee, given how GSLB works: https://searchengineland.com/google-caffeine-now-live-on-one-google-data-center-30697 https://searchengineland.com/google-caffeine-now-live-on-one... I have no clue how DDG works, but I am familiar with the Google search infrastructure as of several years ago and if you told me that the same query should always return the same identical results, I would tell you that you just don't know how Google works, even if you ignore personal results. That made monitoring and catching regressions not fun. It's better than running searches days or weeks apart, but running queries at the same time as they did does not guarantee anything, sorry. Keep also in mind that the average query goes through thousands of machines, according to public statements. Behavior at the long tail can differ for obvious reasons. From a Jeff Dean talk, https://static.googleusercontent.com/media/research.google.com/en//people/jeff/Stanford-DL-Nov-2010.pdf https://static.googleusercontent.com/media/research.google.c... | Some issues: | –Variance: query touches 1000s of machines, not dozens | • e.g. randomized cron jobs caused us trouble for a while | -Availability: 1 or few replicas of each doc’s index data | • Availability of index data when machine failed (esp for important docs): replicate important docs (see also the slides about Universal search, as well as patterns such as the elastic systems and the different tiers)

ryacko 8y ago

You just have to find out the one unique website a particular user always goes to and you can identify that the person is using a computer. Or it is 20 bits of entropy when you go to a website only a thousand other people in the world visit.

saalweachter 8y ago

Frankly, everyone always misunderstands and underestimates the impact of location. Location affects all queries, not just "local" queries. That's literally what is being demonstrated here.

Bartweiss 8y ago
Yes, and I responded quite directly to that. DuckDuckGo's control definitely wasn't adequate, but in return Google's examples ('football' and 'Paris') have unambiguous localization value that's not present for two different Americans searching "immigration". Quite a lot of the variance was link reordering and result changes that didn't have any clear regional aspects, whether at the domain or story level. If localization is the active factor, it's still pretty interesting to know that where I live determines whether to show a Wikipedia link and how to order HuffPo against the Tribune.

puzzle 8y ago

In comparing two people's incognito results, you have no idea if they hit the same search clusters. Were they in the same room? Same ISP? Same geographical region? Same continent? If I issue the same search back to back in different windows, I have a good chance of hitting the same cluster. But I still have no guarantee. Many years ago, SEO people figured that some Google IPs would serve new versions of the ranking algorithms earlier than the rest. Even those were not stable and definitely not a guarantee, given how GSLB works: https://searchengineland.com/google-caffeine-now-live-on-one-google-data-center-30697 https://searchengineland.com/google-caffeine-now-live-on-one... I have no clue how DDG works, but I am familiar with the Google search infrastructure as of several years ago and if you told me that the same query should always return the same identical results, I would tell you that you just don't know how Google works, even if you ignore personal results. That made monitoring and catching regressions not fun. It's better than running searches days or weeks apart, but running queries at the same time as they did does not guarantee anything, sorry. Keep also in mind that the average query goes through thousands of machines, according to public statements. Behavior at the long tail can differ for obvious reasons. From a Jeff Dean talk, https://static.googleusercontent.com/media/research.google.com/en//people/jeff/Stanford-DL-Nov-2010.pdf https://static.googleusercontent.com/media/research.google.c... | Some issues: | –Variance: query touches 1000s of machines, not dozens | • e.g. randomized cron jobs caused us trouble for a while | -Availability: 1 or few replicas of each doc’s index data | • Availability of index data when machine failed (esp for important docs): replicate important docs (see also the slides about Universal search, as well as patterns such as the elastic systems and the different tiers)