8 ms·

Search leakage is not FUD. Google et al., please fix it.

114 points by bjplink 16y ago

andrenotgiant 16y ago
Also posted on site: I agree that the amount of information a well-tagged website can collect on users is frightening, but I don't think that stripping search keyword data from the referrer is the solution. I think Gabriel is going after the wrong thing. Here's why: A good Search Engine will never send a user to a page that isn't textually relevant to the search they entered. In 99.9% of cases, the text they entered is ON the page they hit. So if a user searches for: [SOMETHING CREEPY] they will be hitting a page that already has [SOMETHING CREEPY] published. To put it another way: "Your Keyword data is never going to give a website something it didn't already have. It's just going to reveal what pieces of its content are of interest to you."
- brlewis 16y ago
  It's more complicated than that. See the part of the article about ad networks. On the other hand, I think the gout example was google ads, not an ad network on wikipedia, so hiding referrer info wouldn't have helped.
  - andrenotgiant 16y ago
    agreed. I guess I left out my suggestion for what the problem really is: If you are concerned about Ad Networks having so much data on you, clear your cookies and block cookies from them. Then they will never be able to string together more than one piece of data.
    - subway 16y ago
      You're only bumping the problem from a per-browser (cookie) aggregation to per a machine or household (IP address) aggregation.
    - deleted 16y ago
      [deleted]
    - raganwald 16y ago
      Let's not slip into a false dichotomy. We don't need to argue about whether to block cookies OR block search engine leakage.
  - ultrasaurus 16y ago
    But in this case it's the site choosing to share the search term with the ad network -- and www.medicinenet.com/gout/article.htm (my first result for gout) has a pretty good idea that you searched for gout. It's also a rare enough term (this is my first time ever typing it) that any calculation on statistically improbable phrases will know I searched for "gout", not "article". In most cases the "leakage" is pretty minor from search engine to page, the big leak is from page to ad network.
    - Phil_FL 16y ago
      You are affected if you see ads, using https, using noscript, blocking ads and erasing most of your cookies you are somewhat not vulnerable. Using google the first link that came with 'gout' was wikipedia.
      - ultrasaurus 16y ago
        Still as an advertising demographic "someone researching gout for whatever reason" is more valuable to target than a viewer you know nothing about -- even if half of them are researching the Henry VIII, the other half have a new condition and they need to buy something if only they knew what.
    - joh6nn 16y ago
      "But in this case it's the site choosing to share the search term with the ad network" in the same way that most users probably aren't aware of this, most publishers probably aren't aware as well. i, for one, never considered that the referrer might be passed along to the ad networks i use on my sites. it simply never occurred to me. i definitely didn't choose to share that information with them.
- deleted 16y ago
  [deleted]
benologist 16y ago
"It's unfortunate that DuckDuckGo is preying on people's fears and offering incomplete information in order to garner attention," a company spokeswoman said in an e-mailed statement. It really is impressive that you're on their radar enough to warrant a reaction like that.
- jimboyoungblood 16y ago
  I imagine putting up an anti-Google billboard in SF had something to do with it.
  - benologist 16y ago
    Yup, but it still would have been very easy for them to ignore him and wait to see if there's any non-tech-blog reaction to it.
- seanalltogether 16y ago
  I fear he's losing his goodwill and credibility by taking this angle of attack though. His whole argument really boils down to the fact that advertisers are retarded, not that privacy is being exposed.
  - kleinsch 16y ago
    Agreed. Especially since most cases similar to his argument are retargeting, not query leakage. In his example, if the user actually clicked directly from Google to Wikipedia, Google would be the only one who knew about the user's interest in gout. Google isn't in the business of sharing this information (believe me, it wish they were ;). In most cases that people might assume to be related to this, you search for Timbuk2 bags, click through to their site, then are bombarded with ads all over the internet for Timbuk2 bags. This has nothing to do with search leakage, this is retargeting. Timbuk2 drops advertiser pixels on their site so they can later target those users with advertising. Most advertisers are stupid. They don't have the fancy tech to handle and parse search terms, target users, and display ads. They're probably using RMX or DoubleClick, where you only have the ability to retarget users that have seen certain pixels. They may be using AdSense or AdWords to target queries, but those are using Google's own data, which has nothing to do with search leakage. I think DuckDuckGo rocks, but as someone working in the online advertising industry today, this issue seems manufactured for publicity. This information is useful in theory (and I'm sure a small number of companies are using it) but there are much bigger issues that are getting exploited by everyone.
WillyF 16y ago
Even if search engines stop sending keywords in referral data, the ad networks and webmasters will still be able to piece together your browsing history. Every day there is less and less anonymity on the web, and for the most part people are ok with it. Ten years ago very few people would willingly use their real name online. Facebook changed that. The web as we know it has been built on the assumption that search engines pass along keywords in referrer data. Changing this would have a significant negative impact on a lot of businesses. Considering that most users don't really seem to care about privacy, at least if you judge by actions and not what they say, I don't see why a company like Google would ever stop sending along keyword data to webmasters. They'll piss off webmasters who buy ads from them, and it won't help them increase their share of the search market.
jonknee 16y ago
Referrers aren't passed on to all the elements loading on the page. If you click to nytimes.com from a Google search, the referrer is sent once in the HTTP request to nytimes.com and then your browser makes all the other required requests separately once it gets back the HTML page. When the ads are loaded they don't get your Google referrer, they'll either get nothing or a nytimes.com referrer. You can work around this with JS (which is how Google Analytics works), but it's completely unnecessary for the problems Gabe's talking about... Referrers aren't needed for targeting. On that Gout example, Google knows you researched gout so they can target you with gout ads on sites that run AdSense or DoubleClick (which is a lot of ads). If you visited another site about gout that ran ads from a different network, then they too could target you. The referrer has nothing to do with it, it's what you're requesting. If you don't want targeted advertisements, it's far more effective to use adblock or modify your /etc/hosts file than it is to use DDG.
- rlpb 16y ago
  I know that Javascript can pick up referrers, since Google Analytics gets referrers and uses Javascript and nothing else. Can't advertising networks insert code to pick up referrers in the same way?
  - axod 16y ago
    If the js is running on the main page itself, yes. If it's in an iframe then no.
- benologist 16y ago
  Couldn't the ads access it via JavaScript? http://www.w3schools.com/jsref/prop_doc_referrer.asp http://www.w3schools.com/jsref/prop_doc_referrer.asp Edit: AdSense at least does: var ua=document "&ref=",P(ua.referrer.substring(0,512))
- tghw 16y ago
  While it is true that the requests that load the ads don't include the referrer, any Javascript loaded directly into the page can access the referrer (window.location.orgin). A lot of ad networks work this way, meaning they do have access to those search terms.
bromley 16y ago
"The only reason I've heard to not prevent search leakage is that marketers use Referrer info to do better search engine optimization (SEO). But the information doesn't have to disappear, just the current mechanism of transferring the information in a personally identifiable way." I struggle to see how this could work in a way that's a fraction as useful to webmasters as the current system. Sites that sell things like to tie keywords to conversions. They can learn, for example, that keyword X drives sales, but keyword Y doesn't, and assign resources accordingly. Online businesses become more efficient, and searchers get more of what they want. I think it's largely a good thing all round. My respect goes to DuckDuckGo for coming up with a clever way to differentiate themselves from their competition. However, if the problem is that sites are inadvertently sharing keywords with third-party ad networks, then point the finger at those ad networks, not at Google. Blaming Google makes about as much sense as blaming Firefox, Safari, Internet Explorer and the web in general for sending referrers in the first place.
- dminor 16y ago
  Another issue with this is that it gives Google Analytics a pretty dominant advantage. Suddenly all other analytics packages are shut out of Google search query data, unless Google is benevolent enough to create an API for it.
- calbear81 16y ago
  Thank you for pointing this out bromley. I've been reading Gabriel's responses and in no place does he mention that keyword level data is needed for understanding differences in user behavior based on entry keyword. This might not make a difference if you run a content site, but for any type of commerce site, I don't want to know how "Google" does, I want to break it down to the keyword level and better understand where efforts should be focused. I also don't think that the keywords are shared with 3rd party networks explicitly. I think what's happening instead is that you search for something, you land on a page relevant to that something, and the ad network code is reading the content on the page and assigning a keyword target or theme to your search. For example, you might search for a Ford F-150 and you get to Edmunds and the ads are sold on a "by make/model" basis using ad segmentation so the ad network now can assign your cookie a "Pickup trucks" behavioral tag but it never had to read the referrer header, it was implied.
Matt_Cutts 16y ago
(I'm in all-day training today, so I can't participate on this thread much. Also, this is all my personal opinion.) While Gabe's most recent post was a well-worded statement of his position, my guess is that Google's response was based on the billboard, which says "Google tracks you. We don't." On the website the billboard points to, Google employees are portrayed wearing ski masks and trying to spy on you. That does strike me as trying to a encourage a bit of fear? This is a browser issue that's not specific to Google or even to search engines, but Google is the only company mentioned on donttrack.us until you get to the "more tools" section at the very bottom. Meanwhile, Google is the first (and only) large search engine to offer https to the best of my knowledge. It's a one-character addition to http://www.google.com http://www.google.com for anyone that feels strongly about this topic.
- andrewljohnson 16y ago
  Well, All this FUD has ensured that I'll never try DDG. It's sad really - it's not even a real search engine, just some hack using APIs. And the best marketing Whiny-berg can come up with is half-truths about referer linking. I'll stick with the search engine that has a legion of brilliant programmers working hard to bring me great search results.
  - epi0Bauqu 16y ago
    I'm sorry you feel that way. All I can say is that I tried my hardest in this post to do the opposite of FUD. I believe this is a serious issue and I am trying to get it solved for everyone.
    - axod 16y ago
      You haven't begun to establish people want it "solved". There are already solutions for the people who want them (stop sending referer, adblock, etc). FWIW, I do not want Google to change their behavior in this area.
      - blahedo 16y ago
bobds 16y ago
If you don't want to wait for other people to fix this, there is a handy Firefox addon called No Referrer. It can block referers selectively, either when you click a link on a certain URL, or when you click a link that points to a certain URL. It uses regular expressions so it should be flexible enough. It also blocks referers being sent from localhost/local URLs. I would be interested in trying out an option that only allows referers to be send to the same domain or its subdomains. The interesting part is seeing how many things that option would break. EDIT: Forgot the link. https://addons.mozilla.org/en-US/firefox/addon/no-referrer-misspelled-referer/ https://addons.mozilla.org/en-US/firefox/addon/no-referrer-m...
sogjis 16y ago
I'm using Firefox add-on RefControl, which can remove referrer for 3rd party requests
nkurz 16y ago
While I appreciate that DDG would want to differentiate itself from its competition, if the actual goal is improving user privacy on the internet I don't understand why this is being treated as a Google issue rather than a browser one. Google is an important site, but just one site of many. Wouldn't it make more sense to try to convince browser makers to have HTTP_REFERER turned off by default, either in entirety or for cross-site purposes? It also seems worth noting that if for some reason you wish to continue using Google instead of DDG, and if you are concerned about the potential privacy issues, you can already change your browser not to send the referer header: http://kb.mozillazine.org/Network.http.sendRefererHeader http://kb.mozillazine.org/Network.http.sendRefererHeader https://chrome.google.com/extensions/detail/dkpkjedlegmelkogpgamcaemgbanohip https://chrome.google.com/extensions/detail/dkpkjedlegmelkog...
- othermaciej 16y ago
  Not sending the Referer header at all can break some sites, probably more than is acceptable to do by default. But stripping the query part of the Referer header might be reasonable. Probably the main side effect of that would be to make Google mad. (P.S. the header is called "Referer", not "HTTP_REFERER").
  - nkurz 16y ago
    Which sites would break, and why? (genuine question, not contrariness) I've always assumed that cross-site Referer is sufficiently brittle that one cannot depend on it, since it's isn't there if one enters a URL by hand or arrives via a redirect. And while I like it for use within a site, it seems that Cookies have taken over for most uses. Sorry about the sloppiness with HTTP_REFERER vs Referer. You are correct --- I tend to think of it from the CGI point of view rather than browser. Browser sends Referer as an HTTP header, which web servers commonly set in the environment as HTTP_REFERER. Thus the question should be "Why not have browsers default to not sending the HTTP Referer header?"
tony_landis 16y ago
The author is singling out out one company and saying they should be doing things differently than the rest of the web, because of what 3rd parties can do as a result. Why not go after the advertisers if they are the real miscreants? FUD! There is absolutely no reason that Google should break the web to pacify this guy.
- gloob 16y ago
  Taking his suggestions would no more "break the web" than Craigslist broke paper. It would harm a common business model, sure, but I have yet to see a good reason for me to care about the business models of web companies any more than I care about the business models of newspapers or record companies.
- gergles 16y ago
  I wish there was a way to flag comments that obviously did not read the article, as there's an entire paragraph devoted to "omg you're just attacking Google you meaniehead!"
jemfinch 16y ago
If you care about search leakage, turn the Referer header off in your browser. Problem solved. Why is it any website's job to change the way HTTP is designed to work?
- blub 16y ago
  Ok. Now how do I turn off the data mining?
  - jonknee 16y ago
    Don't send requests to domains you believe are mining your data.
    - gloob 16y ago
      Or to domains that might log your requests and later be bought out by a company that mines your data. Or to domains that might log your requests and turn around and sell the info to others. Or to domains that might log your requests and then be cracked. Oh wait. That describes half the fucking sites on the web.
    - blub 16y ago
      I try to avoid it, though it is hard these days since so many companies do it. So besides that I'm also trying to convince people that they shouldn't put up with it.
axod 16y ago
Constantly attacking Google over something the vast majority of users don't care about seems like a bad idea. I was ready to try duckduckgo if it could give me the results I wanted (Despite the hugely irritating UI and infinite scroll). But the constant attacking Google seems bad business to me. It IS FUD. Google doesn't track you. Your browser sends a referer header, which it has done since the dawn of time. Who cares? flagged. I think you're going to lose a lot of goodwill Gabriel.
- epi0Bauqu 16y ago
  This is my last post on the subject. I felt that my position was being read unfairly, and I wanted to set the record straight. I apologize if it did not come off that way as it was clearly not the intention. I truly believe this is an unnecessary leaking of personal information. And I address the browser argument directly in the post, as well as the argument that no one cares.
  - axod 16y ago
    Looking forward to some posts from you on improved UI, removal of infinite scroll, and improved search results :)
  - joh6nn 16y ago
    Gabriel, there seems to be some confusion here in the thread about "breaking HTTP_REFERRER" or in some way changing the current referrer behavior, which was not how i understood your post. can you confirm quickly that you are proposing that search engines sanitize/anonymize referrer data, and not that they somehow change the referrer behavior?
    - epi0Bauqu 16y ago
      I guess it depends what you mean by "break." I meant what you just said, i.e. just drop the search terms.
- blub 16y ago
  The Google TOS clearly states that they have the right to analyze your content in order for them to provide ads. This includes Gmail, Docs and whatever else you may use. When it comes to search they save your search history for customized searches. Let me put it this way: they don't target those ads so precisely by not having any information on you. On the contrary, they have lots of info. How safe and how anonymous that info is, that's up for debate.
JonnieCache 16y ago
https://encrypted.google.com https://encrypted.google.com SSL pages prevent referer headers from being sent. Easy. The country specific pages dont have equivalents, so no encrypted.google.co.uk, but you can get the same effect using the gl parameter in the URL, so the url for a UK search would be: https://encrypted.google.com/search?gl=uk&q=foo https://encrypted.google.com/search?gl=uk&q=foo Get your list of valid country codes here: http://www.google.com/cse/docs/resultsxml.html#countryCodes http://www.google.com/cse/docs/resultsxml.html#countryCodes
- JonnieCache 16y ago
  Too late to edit but I should point out that this also handily prevents governments or ISPs from viewing your queries, or anyone else for that matter. Except in cases of MITM attacks obviously.
random42 16y ago
I use DDG as my primary search engine and plan to continue, but I am really getting tired of this aggresive/attacking marketing approach of gaberial.
armandososa 16y ago
I don't feel qualified to say whether this is FUD or not, but it certainly imposed some fear on me. I don't know what a Gout is and now I'm afraid to search for it.
- joh6nn 16y ago
  From the American Heritage Dictionary: "Gout: (n) A disturbance of uric-acid metabolism occurring chiefly in males, characterized by painful inflammation of the joints, especially of the feet and hands, and arthritic attacks resulting from elevated levels of uric acid in the blood and the deposition of urate crystals around the joints. The condition can become chronic and result in deformity."
- tdfx 16y ago
  The gout example in the article strikes me as very odd, since the only site that should've received the referrer with the search terms was wikipedia. Therefore we've left to conclude that either Google is targeting ads in their AdSense network based on search terms (which is outside the scope of Gabriel's argument) or Wikipedia is passing search terms to ad networks that for some reason it doesn't display ads from. Equally likely is that this concerned user clicked on a different health-related site with an ad network that classified him according to that site's content or stated category -- there's simply no evidence that the aggregation of search terms happened. Note: I'm not saying the story couldn't be true, just pointing out that no technical evidence has been presented to rule out the other possibilities.
ohyes 16y ago
Perhaps I'm not thinking malevolently enough, but in what situation would the search terms that I used be enough to invade my privacy? Presumably, the content of the site is related to whatever you searched for (otherwise you wouldn't click on the link). If you are willing to click the link and go to the site, the site will most likely have some idea of why you are there, and what you are interested in, regardless of the referrer headers (because, you know, the site is hosting the content that you are reading). It seems that if I am willing to visit the site at all, I should also be willing to disclose trivial information like this. So I'm not sure why I should care. Saying that this is not disclosed also seems a little disingenuous. Referrer headers are pretty standard. If you have a problem with Google doing this, you also have a problem with pretty much every other site that uses hyper-links. It seems that there is a lot of useful semantic information that could be gathered by being able to identify which documents reference your document. Eliminating referrer headers seems like it would be a net loss (pun not intended).
- joh6nn 16y ago
  yep, you're not being malicious enough :) the issue is not whether the destination site receives the search terms (and indeed, Gabriel suggests that they should continue to do so, either through the GWT, or some other method). the issues is that currently, any advertising networks in use by the destination site also receive the search terms, via the same mechanism: the referrer. that's the crux of the issue. while the destination site can't follow your traffic once you leave it, the ad networks, because of their large user base, frequently can. they can begin to build a much more thorough profile of who you are and what you are searching for than anyone single destination site could. whether that's an invasion of privacy is your call, but to many people it is. currently, they're simply unaware that it's happening.
  - ohyes 16y ago
    Thanks for explaining, I can now see how this might be a problem. Through this same mechanism wouldn't the advertising networks be privy to the content of the sites that I am visiting? It seems that even if we eliminate this, we still have issues with advertisers being able to track and create a profile based on the content of the websites you are visiting. The headers do seem to create a direct link between a given search and a set of visited sites, but can't things like cookies and tracking pixels be used to the same effect? Possibly then using NLP to figure out the most important words on the page? Or the SEO terms that the website uses to get picked up by the search engine? If you are going to let an advertiser post content on your site, it seems to me that it would be very difficult to keep said advertiser from tracking your users. If the user uses Adblocking software or otherwise blocks the advertisers' sub-domains, does the advertiser still receive the referral headers?
tmsh 16y ago
In Gabriel's defense, I'd say there's something very antithetical between FUD and a detailed description on how to fix the (alleged) problem. I don't know of any other FUD campaign in which a simple solution was provided -- one which won't directly benefit the entity raising the objection. It's Google, et al.'s decision which way they want to go -- whether they take his advice or ignore it. But one could excuse the billboard potentially to a person trying to highlight that this is a big issue. But again, this seems quite different from an incumbent that is trying to cast doubt via obfuscation, which has usually been the case in FUD...
- jacquesm 16y ago
  I see the billboard as nothing more than a prank that is now pulled out to beat down the discussion about the actual subject matter.
Zakuzaa 16y ago
Anybody paying attention to http://www.techmeme.com/110124/p25#a110124p25 http://www.techmeme.com/110124/p25#a110124p25 ?
tdfx 16y ago
There aren't enough people at the FTC to read the complaints that would flood in if Google changed this and Google Analytics became the only tracking platform that could do SEO keyword analysis.
- jacquesm 16y ago
  I don't see any reason why analytics should have access to that information when other tracking platforms would not. Chinese walls should take care of that. And if those are not in place then google has bigger problems.

114 points by bjplink 16y ago

Also posted on site: I agree that the amount of information a well-tagged website can collect on users is frightening, but I don't think that stripping search keyword data from the referrer is the solution. I think Gabriel is going after the wrong thing. Here's why: A good Search Engine will never send a user to a page that isn't textually relevant to the search they entered. In 99.9% of cases, the text they entered is ON the page they hit. So if a user searches for: [SOMETHING CREEPY] they will be hitting a page that already has [SOMETHING CREEPY] published. To put it another way: "Your Keyword data is never going to give a website something it didn't already have. It's just going to reveal what pieces of its content are of interest to you."

brlewis 16y ago
It's more complicated than that. See the part of the article about ad networks. On the other hand, I think the gout example was google ads, not an ad network on wikipedia, so hiding referrer info wouldn't have helped.
- andrenotgiant 16y ago
  agreed. I guess I left out my suggestion for what the problem really is: If you are concerned about Ad Networks having so much data on you, clear your cookies and block cookies from them. Then they will never be able to string together more than one piece of data.
  - subway 16y ago
    You're only bumping the problem from a per-browser (cookie) aggregation to per a machine or household (IP address) aggregation.
  - deleted 16y ago
    [deleted]
  - raganwald 16y ago
    Let's not slip into a false dichotomy. We don't need to argue about whether to block cookies OR block search engine leakage.
- ultrasaurus 16y ago
  But in this case it's the site choosing to share the search term with the ad network -- and www.medicinenet.com/gout/article.htm (my first result for gout) has a pretty good idea that you searched for gout. It's also a rare enough term (this is my first time ever typing it) that any calculation on statistically improbable phrases will know I searched for "gout", not "article". In most cases the "leakage" is pretty minor from search engine to page, the big leak is from page to ad network.
  - Phil_FL 16y ago
    You are affected if you see ads, using https, using noscript, blocking ads and erasing most of your cookies you are somewhat not vulnerable. Using google the first link that came with 'gout' was wikipedia.
    - ultrasaurus 16y ago
      Still as an advertising demographic "someone researching gout for whatever reason" is more valuable to target than a viewer you know nothing about -- even if half of them are researching the Henry VIII, the other half have a new condition and they need to buy something if only they knew what.
  - joh6nn 16y ago
    "But in this case it's the site choosing to share the search term with the ad network" in the same way that most users probably aren't aware of this, most publishers probably aren't aware as well. i, for one, never considered that the referrer might be passed along to the ad networks i use on my sites. it simply never occurred to me. i definitely didn't choose to share that information with them.
deleted 16y ago
[deleted]

benologist 16y ago

"It's unfortunate that DuckDuckGo is preying on people's fears and offering incomplete information in order to garner attention," a company spokeswoman said in an e-mailed statement. It really is impressive that you're on their radar enough to warrant a reaction like that.

jimboyoungblood 16y ago
I imagine putting up an anti-Google billboard in SF had something to do with it.
- benologist 16y ago
  Yup, but it still would have been very easy for them to ignore him and wait to see if there's any non-tech-blog reaction to it.
seanalltogether 16y ago
I fear he's losing his goodwill and credibility by taking this angle of attack though. His whole argument really boils down to the fact that advertisers are retarded, not that privacy is being exposed.
- kleinsch 16y ago
  Agreed. Especially since most cases similar to his argument are retargeting, not query leakage. In his example, if the user actually clicked directly from Google to Wikipedia, Google would be the only one who knew about the user's interest in gout. Google isn't in the business of sharing this information (believe me, it wish they were ;). In most cases that people might assume to be related to this, you search for Timbuk2 bags, click through to their site, then are bombarded with ads all over the internet for Timbuk2 bags. This has nothing to do with search leakage, this is retargeting. Timbuk2 drops advertiser pixels on their site so they can later target those users with advertising. Most advertisers are stupid. They don't have the fancy tech to handle and parse search terms, target users, and display ads. They're probably using RMX or DoubleClick, where you only have the ability to retarget users that have seen certain pixels. They may be using AdSense or AdWords to target queries, but those are using Google's own data, which has nothing to do with search leakage. I think DuckDuckGo rocks, but as someone working in the online advertising industry today, this issue seems manufactured for publicity. This information is useful in theory (and I'm sure a small number of companies are using it) but there are much bigger issues that are getting exploited by everyone.

WillyF 16y ago

Even if search engines stop sending keywords in referral data, the ad networks and webmasters will still be able to piece together your browsing history. Every day there is less and less anonymity on the web, and for the most part people are ok with it. Ten years ago very few people would willingly use their real name online. Facebook changed that. The web as we know it has been built on the assumption that search engines pass along keywords in referrer data. Changing this would have a significant negative impact on a lot of businesses. Considering that most users don't really seem to care about privacy, at least if you judge by actions and not what they say, I don't see why a company like Google would ever stop sending along keyword data to webmasters. They'll piss off webmasters who buy ads from them, and it won't help them increase their share of the search market.

jonknee 16y ago

Referrers aren't passed on to all the elements loading on the page. If you click to nytimes.com from a Google search, the referrer is sent once in the HTTP request to nytimes.com and then your browser makes all the other required requests separately once it gets back the HTML page. When the ads are loaded they don't get your Google referrer, they'll either get nothing or a nytimes.com referrer. You can work around this with JS (which is how Google Analytics works), but it's completely unnecessary for the problems Gabe's talking about... Referrers aren't needed for targeting. On that Gout example, Google knows you researched gout so they can target you with gout ads on sites that run AdSense or DoubleClick (which is a lot of ads). If you visited another site about gout that ran ads from a different network, then they too could target you. The referrer has nothing to do with it, it's what you're requesting. If you don't want targeted advertisements, it's far more effective to use adblock or modify your /etc/hosts file than it is to use DDG.

rlpb 16y ago
I know that Javascript can pick up referrers, since Google Analytics gets referrers and uses Javascript and nothing else. Can't advertising networks insert code to pick up referrers in the same way?
- axod 16y ago
  If the js is running on the main page itself, yes. If it's in an iframe then no.
benologist 16y ago
Couldn't the ads access it via JavaScript? http://www.w3schools.com/jsref/prop_doc_referrer.asp http://www.w3schools.com/jsref/prop_doc_referrer.asp Edit: AdSense at least does: var ua=document "&ref=",P(ua.referrer.substring(0,512))
tghw 16y ago
While it is true that the requests that load the ads don't include the referrer, any Javascript loaded directly into the page can access the referrer (window.location.orgin). A lot of ad networks work this way, meaning they do have access to those search terms.

bromley 16y ago

"The only reason I've heard to not prevent search leakage is that marketers use Referrer info to do better search engine optimization (SEO). But the information doesn't have to disappear, just the current mechanism of transferring the information in a personally identifiable way." I struggle to see how this could work in a way that's a fraction as useful to webmasters as the current system. Sites that sell things like to tie keywords to conversions. They can learn, for example, that keyword X drives sales, but keyword Y doesn't, and assign resources accordingly. Online businesses become more efficient, and searchers get more of what they want. I think it's largely a good thing all round. My respect goes to DuckDuckGo for coming up with a clever way to differentiate themselves from their competition. However, if the problem is that sites are inadvertently sharing keywords with third-party ad networks, then point the finger at those ad networks, not at Google. Blaming Google makes about as much sense as blaming Firefox, Safari, Internet Explorer and the web in general for sending referrers in the first place.

dminor 16y ago
Another issue with this is that it gives Google Analytics a pretty dominant advantage. Suddenly all other analytics packages are shut out of Google search query data, unless Google is benevolent enough to create an API for it.
calbear81 16y ago
Thank you for pointing this out bromley. I've been reading Gabriel's responses and in no place does he mention that keyword level data is needed for understanding differences in user behavior based on entry keyword. This might not make a difference if you run a content site, but for any type of commerce site, I don't want to know how "Google" does, I want to break it down to the keyword level and better understand where efforts should be focused. I also don't think that the keywords are shared with 3rd party networks explicitly. I think what's happening instead is that you search for something, you land on a page relevant to that something, and the ad network code is reading the content on the page and assigning a keyword target or theme to your search. For example, you might search for a Ford F-150 and you get to Edmunds and the ads are sold on a "by make/model" basis using ad segmentation so the ad network now can assign your cookie a "Pickup trucks" behavioral tag but it never had to read the referrer header, it was implied.

Matt_Cutts 16y ago

(I'm in all-day training today, so I can't participate on this thread much. Also, this is all my personal opinion.) While Gabe's most recent post was a well-worded statement of his position, my guess is that Google's response was based on the billboard, which says "Google tracks you. We don't." On the website the billboard points to, Google employees are portrayed wearing ski masks and trying to spy on you. That does strike me as trying to a encourage a bit of fear? This is a browser issue that's not specific to Google or even to search engines, but Google is the only company mentioned on donttrack.us until you get to the "more tools" section at the very bottom. Meanwhile, Google is the first (and only) large search engine to offer https to the best of my knowledge. It's a one-character addition to http://www.google.com http://www.google.com for anyone that feels strongly about this topic.

andrewljohnson 16y ago
Well, All this FUD has ensured that I'll never try DDG. It's sad really - it's not even a real search engine, just some hack using APIs. And the best marketing Whiny-berg can come up with is half-truths about referer linking. I'll stick with the search engine that has a legion of brilliant programmers working hard to bring me great search results.
- epi0Bauqu 16y ago
  I'm sorry you feel that way. All I can say is that I tried my hardest in this post to do the opposite of FUD. I believe this is a serious issue and I am trying to get it solved for everyone.
  - axod 16y ago
    You haven't begun to establish people want it "solved". There are already solutions for the people who want them (stop sending referer, adblock, etc). FWIW, I do not want Google to change their behavior in this area.
    - blahedo 16y ago

bobds 16y ago

If you don't want to wait for other people to fix this, there is a handy Firefox addon called No Referrer. It can block referers selectively, either when you click a link on a certain URL, or when you click a link that points to a certain URL. It uses regular expressions so it should be flexible enough. It also blocks referers being sent from localhost/local URLs. I would be interested in trying out an option that only allows referers to be send to the same domain or its subdomains. The interesting part is seeing how many things that option would break. EDIT: Forgot the link. https://addons.mozilla.org/en-US/firefox/addon/no-referrer-misspelled-referer/ https://addons.mozilla.org/en-US/firefox/addon/no-referrer-m...

sogjis 16y ago

I'm using Firefox add-on RefControl, which can remove referrer for 3rd party requests

nkurz 16y ago

While I appreciate that DDG would want to differentiate itself from its competition, if the actual goal is improving user privacy on the internet I don't understand why this is being treated as a Google issue rather than a browser one. Google is an important site, but just one site of many. Wouldn't it make more sense to try to convince browser makers to have HTTP_REFERER turned off by default, either in entirety or for cross-site purposes? It also seems worth noting that if for some reason you wish to continue using Google instead of DDG, and if you are concerned about the potential privacy issues, you can already change your browser not to send the referer header: http://kb.mozillazine.org/Network.http.sendRefererHeader http://kb.mozillazine.org/Network.http.sendRefererHeader https://chrome.google.com/extensions/detail/dkpkjedlegmelkogpgamcaemgbanohip https://chrome.google.com/extensions/detail/dkpkjedlegmelkog...

othermaciej 16y ago
Not sending the Referer header at all can break some sites, probably more than is acceptable to do by default. But stripping the query part of the Referer header might be reasonable. Probably the main side effect of that would be to make Google mad. (P.S. the header is called "Referer", not "HTTP_REFERER").
- nkurz 16y ago
  Which sites would break, and why? (genuine question, not contrariness) I've always assumed that cross-site Referer is sufficiently brittle that one cannot depend on it, since it's isn't there if one enters a URL by hand or arrives via a redirect. And while I like it for use within a site, it seems that Cookies have taken over for most uses. Sorry about the sloppiness with HTTP_REFERER vs Referer. You are correct --- I tend to think of it from the CGI point of view rather than browser. Browser sends Referer as an HTTP header, which web servers commonly set in the environment as HTTP_REFERER. Thus the question should be "Why not have browsers default to not sending the HTTP Referer header?"

tony_landis 16y ago

The author is singling out out one company and saying they should be doing things differently than the rest of the web, because of what 3rd parties can do as a result. Why not go after the advertisers if they are the real miscreants? FUD! There is absolutely no reason that Google should break the web to pacify this guy.

gloob 16y ago
Taking his suggestions would no more "break the web" than Craigslist broke paper. It would harm a common business model, sure, but I have yet to see a good reason for me to care about the business models of web companies any more than I care about the business models of newspapers or record companies.
gergles 16y ago
I wish there was a way to flag comments that obviously did not read the article, as there's an entire paragraph devoted to "omg you're just attacking Google you meaniehead!"

jemfinch 16y ago

If you care about search leakage, turn the Referer header off in your browser. Problem solved. Why is it any website's job to change the way HTTP is designed to work?

blub 16y ago
Ok. Now how do I turn off the data mining?
- jonknee 16y ago
  Don't send requests to domains you believe are mining your data.
  - gloob 16y ago
    Or to domains that might log your requests and later be bought out by a company that mines your data. Or to domains that might log your requests and turn around and sell the info to others. Or to domains that might log your requests and then be cracked. Oh wait. That describes half the fucking sites on the web.
  - blub 16y ago
    I try to avoid it, though it is hard these days since so many companies do it. So besides that I'm also trying to convince people that they shouldn't put up with it.

axod 16y ago

Constantly attacking Google over something the vast majority of users don't care about seems like a bad idea. I was ready to try duckduckgo if it could give me the results I wanted (Despite the hugely irritating UI and infinite scroll). But the constant attacking Google seems bad business to me. It IS FUD. Google doesn't track you. Your browser sends a referer header, which it has done since the dawn of time. Who cares? flagged. I think you're going to lose a lot of goodwill Gabriel.

epi0Bauqu 16y ago
This is my last post on the subject. I felt that my position was being read unfairly, and I wanted to set the record straight. I apologize if it did not come off that way as it was clearly not the intention. I truly believe this is an unnecessary leaking of personal information. And I address the browser argument directly in the post, as well as the argument that no one cares.
- axod 16y ago
  Looking forward to some posts from you on improved UI, removal of infinite scroll, and improved search results :)
- joh6nn 16y ago
  Gabriel, there seems to be some confusion here in the thread about "breaking HTTP_REFERRER" or in some way changing the current referrer behavior, which was not how i understood your post. can you confirm quickly that you are proposing that search engines sanitize/anonymize referrer data, and not that they somehow change the referrer behavior?
  - epi0Bauqu 16y ago
    I guess it depends what you mean by "break." I meant what you just said, i.e. just drop the search terms.
blub 16y ago
The Google TOS clearly states that they have the right to analyze your content in order for them to provide ads. This includes Gmail, Docs and whatever else you may use. When it comes to search they save your search history for customized searches. Let me put it this way: they don't target those ads so precisely by not having any information on you. On the contrary, they have lots of info. How safe and how anonymous that info is, that's up for debate.

JonnieCache 16y ago

https://encrypted.google.com https://encrypted.google.com SSL pages prevent referer headers from being sent. Easy. The country specific pages dont have equivalents, so no encrypted.google.co.uk, but you can get the same effect using the gl parameter in the URL, so the url for a UK search would be: https://encrypted.google.com/search?gl=uk&q=foo https://encrypted.google.com/search?gl=uk&q=foo Get your list of valid country codes here: http://www.google.com/cse/docs/resultsxml.html#countryCodes http://www.google.com/cse/docs/resultsxml.html#countryCodes

JonnieCache 16y ago
Too late to edit but I should point out that this also handily prevents governments or ISPs from viewing your queries, or anyone else for that matter. Except in cases of MITM attacks obviously.

random42 16y ago

I use DDG as my primary search engine and plan to continue, but I am really getting tired of this aggresive/attacking marketing approach of gaberial.

armandososa 16y ago

I don't feel qualified to say whether this is FUD or not, but it certainly imposed some fear on me. I don't know what a Gout is and now I'm afraid to search for it.

joh6nn 16y ago
From the American Heritage Dictionary: "Gout: (n) A disturbance of uric-acid metabolism occurring chiefly in males, characterized by painful inflammation of the joints, especially of the feet and hands, and arthritic attacks resulting from elevated levels of uric acid in the blood and the deposition of urate crystals around the joints. The condition can become chronic and result in deformity."
tdfx 16y ago
The gout example in the article strikes me as very odd, since the only site that should've received the referrer with the search terms was wikipedia. Therefore we've left to conclude that either Google is targeting ads in their AdSense network based on search terms (which is outside the scope of Gabriel's argument) or Wikipedia is passing search terms to ad networks that for some reason it doesn't display ads from. Equally likely is that this concerned user clicked on a different health-related site with an ad network that classified him according to that site's content or stated category -- there's simply no evidence that the aggregation of search terms happened. Note: I'm not saying the story couldn't be true, just pointing out that no technical evidence has been presented to rule out the other possibilities.

ohyes 16y ago

Perhaps I'm not thinking malevolently enough, but in what situation would the search terms that I used be enough to invade my privacy? Presumably, the content of the site is related to whatever you searched for (otherwise you wouldn't click on the link). If you are willing to click the link and go to the site, the site will most likely have some idea of why you are there, and what you are interested in, regardless of the referrer headers (because, you know, the site is hosting the content that you are reading). It seems that if I am willing to visit the site at all, I should also be willing to disclose trivial information like this. So I'm not sure why I should care. Saying that this is not disclosed also seems a little disingenuous. Referrer headers are pretty standard. If you have a problem with Google doing this, you also have a problem with pretty much every other site that uses hyper-links. It seems that there is a lot of useful semantic information that could be gathered by being able to identify which documents reference your document. Eliminating referrer headers seems like it would be a net loss (pun not intended).

joh6nn 16y ago
yep, you're not being malicious enough :) the issue is not whether the destination site receives the search terms (and indeed, Gabriel suggests that they should continue to do so, either through the GWT, or some other method). the issues is that currently, any advertising networks in use by the destination site also receive the search terms, via the same mechanism: the referrer. that's the crux of the issue. while the destination site can't follow your traffic once you leave it, the ad networks, because of their large user base, frequently can. they can begin to build a much more thorough profile of who you are and what you are searching for than anyone single destination site could. whether that's an invasion of privacy is your call, but to many people it is. currently, they're simply unaware that it's happening.
- ohyes 16y ago
  Thanks for explaining, I can now see how this might be a problem. Through this same mechanism wouldn't the advertising networks be privy to the content of the sites that I am visiting? It seems that even if we eliminate this, we still have issues with advertisers being able to track and create a profile based on the content of the websites you are visiting. The headers do seem to create a direct link between a given search and a set of visited sites, but can't things like cookies and tracking pixels be used to the same effect? Possibly then using NLP to figure out the most important words on the page? Or the SEO terms that the website uses to get picked up by the search engine? If you are going to let an advertiser post content on your site, it seems to me that it would be very difficult to keep said advertiser from tracking your users. If the user uses Adblocking software or otherwise blocks the advertisers' sub-domains, does the advertiser still receive the referral headers?

tmsh 16y ago

In Gabriel's defense, I'd say there's something very antithetical between FUD and a detailed description on how to fix the (alleged) problem. I don't know of any other FUD campaign in which a simple solution was provided -- one which won't directly benefit the entity raising the objection. It's Google, et al.'s decision which way they want to go -- whether they take his advice or ignore it. But one could excuse the billboard potentially to a person trying to highlight that this is a big issue. But again, this seems quite different from an incumbent that is trying to cast doubt via obfuscation, which has usually been the case in FUD...

jacquesm 16y ago
I see the billboard as nothing more than a prank that is now pulled out to beat down the discussion about the actual subject matter.

Zakuzaa 16y ago

Anybody paying attention to http://www.techmeme.com/110124/p25#a110124p25 http://www.techmeme.com/110124/p25#a110124p25 ?

tdfx 16y ago

There aren't enough people at the FTC to read the complaints that would flood in if Google changed this and Google Analytics became the only tracking platform that could do SEO keyword analysis.

jacquesm 16y ago
I don't see any reason why analytics should have access to that information when other tracking platforms would not. Chinese walls should take care of that. And if those are not in place then google has bigger problems.