9 ms·

Where did all the HTTP referrers go?

110 points by Smerity 13y ago

warcode 13y ago
I already hide/change my referrer through browser extensions, though I guess this might finally get those few sites who have functionality based on refs to stop using it.
- Smerity 13y ago
  Whilst I still think referrers should be used where appropriate, there are certainly places or reasons you want to nix them. HTTPS not passing along referrers by default is a sensible decision, as is the addition of <meta name="referrer" content="never"> for sites stuck in HTTP that may want to go under the radar. I still feel that removing referrers entirely destroys many useful tools and analytics that we've traditionally been able to use. It removes the core way in which we understand connections across the Internet. By removing referrers, the best we can do is use link graphs, falling back to the original PageRank algorithm where we assume people are random bots that click on one of the links on the page. Edit: Can't reply to you hnriot due to comment depth limit. My reference to PageRank is as links and backlinks could be used as a poor referrer substitute, though they aren't currently used as it's a lot more work and less accurate. I'm simply saying that, in the event that referrers all disappeared tomorrow, you'd see normal websites trying to estimate where their traffic comes from by using a PageRank inspired algorithm, or more naively by looking at who links where.
  - hnriot 13y ago
    Referrers have nothing to do with pagerank. It's used in website analytics of course, and, in general is a very bad idea leaking information between websites. Search engines don't see your site's referrer data so the association you tried to create with pagerank is misleading.
    - encoderer 13y ago
      I, for one, didn't think his argument was remotely misleading. I did, however, think that bringing PageRank into the argument was ill advised and probably detrimental to his goal of advocacy through education. It just confused things, introduced another rabbit-hole concept. If the author thought that mentioning PageRank would lend credibility to the 'link counting' / 'random link clicking bots' foregone conclusion he setup., he was right. It did. So link the text to a footnote referencing PageRank and stay on topic.
anonymouz 13y ago
Referers are a privacy trade-off though, and personally I have never seen a good reason why the desire of some website to track where I am coming from should trump my desire to not broadcast this information to the world. Therefore I like to switch of referers anyway.
- claudius 13y ago
  I am doing the same. Fortunately, the websites requiring the referer to be set are getting fewer and fewer – some years ago, I regularly got placeholder images ‘THIS IMAGE WAS STOLEN FROM XYZ’ when browsing XYZ without referers.
  - joosters 13y ago
    This is one of the good uses of HTTP referrers! I host a small, very low traffic website. One day, the bandwidth shoots through the roof and stays high. The reason? One of the images on a page got added to the .sig of someone in a popular forum. Suddenly thousands of people are fetching the image. The solution was to filter by referrer header, letting the image be seen by visitors to the actual page, but linking from other sites gets blocked. Note that usually it's best to allow requests that have no referrer header at all, otherwise you'll be blocking some legitimate viewers of your site. End result: bandwidth back down to the usual, tiny levels. It's not that I cared about people copying the images, I just didn't want to foot the bill for the traffic!
    - graue 13y ago
      I think you misunderstand — this seems to have been claudius' point: > Note that usually it's best to allow requests that have no referrer header at all, otherwise you'll be blocking some legitimate viewers of your site.
      - joosters 13y ago
        It was probably a badly-placed reply; my point was to illustrate a use for referrer headers and to state that sites blocking based on the lack of a referrer were doing it wrong.
lucb1e 13y ago
Oh I did not know this. Very good to know indeed! I too am missing referrers from websites, and since my website also supports https I suppose others are missing them from me as well. This way I can make the internet a very tiny bit better ^^.
mooism2 13y ago
So my understanding is that the default behaviour is: 1. Follow link from https://example.org https://example.org to http://example.com http://example.com --- referrer is not sent 2. Follow link from https://example.org https://example.org to https://example.com https://example.com --- referrer is sent I don't understand how the same referrer can be too sensitive to be sent as plaintext, but harmless enough to be passed to a not-necessarily-trusted third party.
- deleted 13y ago
  [deleted]
- kyrias 13y ago
  It's more about who can see it along the way than it is about who receives the request.
  - ctz 13y ago
    I'm willing to bet that it gets sent even if the second HTTPS connection provides no effective security. (If this is the protocol we're building economies on these days, I feel computer security is going to get much worse before it gets better...)
    - joosters 13y ago
      If your browser is communicating on the second HTTPS connection, that pretty much implies that you are happy with its security levels, right? After all, if you weren't (e.g. bad certificates, etc), your browser wouldn't have established & confirmed the secure connection, HTTP referrer headers or not.
      - ctz 13y ago
        You're right, of course. Let me expand my argument a little. We have two security domains, associated with the first and second server. The first server has sensitive URLs, the second receives these in the referrer header. In making the first connection, the client and first server get to make some security policy decisions: the crypto used in securing the connection, the availability of countermeasures against known TLS protocol vulnerabilities, client and server authentication methods, etc. These parameters are complex, and you end up with a connection with some security level given some attacker model. If you think about it really hard, you can come up with a quantitive estimate of what security you might get from the connection in terms of attacker work -- anywhere from zero (trivially broken) to 256-bit security (very good). Assume our first connection has a 128-bit security level and has countermeasures for TLS vulnerabilities. This is good going. Now we serve over this connection a https link, which the user clicks. Unfortunately the client and/or second server is poorly configured and we only manage (for the sake of argument) a 40-bit security level and no TLS vulnerability countermeasures. Now we've reduced portions of the first connection to the security level offerred by the second connection. This is really very surprising.
dlsym 13y ago
But why is that a bad thing? Just count your page impressions to see if you blogcontent is popular. What websites your visitors consume is IMHO not your business.
- josteink 13y ago
  > What websites your visitors consume is IMHO not your business. But if your product is the subject of discussion on another page which links to yours, you may want to be able to join the discussion. As a site owner, I don't think it's unreasonable to ask my visitors how and where they found out about my website.
- jonknee 13y ago
  It's two very different things if the sudden spike in page views is because of nytimes.com or reddit.com.
w00kie 13y ago
Thanks for the info, I was not aware of this meta tag. My blog is all HTTPS (with SPDY) and I added content="origin" to all visitor facing pages, leaving the admin pages dark.
cnahr 13y ago
When I recently tried to purge my server logs of referrer spam, I found that a whopping 80% of all visitors had either no referrer or referred from my own website. About a third of the rest were spam. So judging from my small sample, referrer headers are largely useless except as spam machines. They should probably be removed from HTTP anyway.
kmasters 13y ago
I fail to see why referrer is part of any spec. Referrer is a 90's concoction when we were all so naive of future uses. I'm sure it's made a lot of porn sites some money, but affiliate marketing is not the purpose of web standards. As far as analytics its all done with tracking cookies now, which is another issue that needs to be addressed, but Im not going to miss referer should it ever really go away.
- jacques_chester 13y ago
  My honours project was on the problems of tracking users visiting different websites using cookies. The problems are numerous. I think I went through 9 revisions of the "naive protocol" (including, at one point, ditching the referer header in favour of HTTPS). Subsequently I realised that my tracking protocol was broken anyhow. My conclusion is that there's no reliable way to track users visiting multiple websites using the standard features of HTML/JS/HTTP in the face of malicious users or publishers. You have to fall back on traffic analysis. I developed a successor technology which works better in many respects (but not all). As it's the subject of a current patent application I can't really go into much detail.
  - mayank 13y ago
    I hope you've looked at evercookie before you go about filing patents on user tracking...
    - jacques_chester 13y ago
      It's a quite different scheme.
- Angostura 13y ago
  I see it rather differently. Referrer is a 1990s concoction based on the idea that the Web is a collaborative effort that we all engage in. I don't personally have any problem in a Web site knowing where I arrive from in most cases. It's harmless him of information (in most cases, as I say) which makes the Web site owner's life a little better.
kvi 13y ago
Yeah, that's a long known thing. I use HTTPS on parsebin.com specifically for stripping out referrers.
nathell 13y ago
This is how I discovered HN went HTTPS some time ago. I never noticed until now.
- sp332 13y ago
  HN started using SPDY recently, and SPDY requires HTTPS. https://news.ycombinator.com/item?id=5660797 https://news.ycombinator.com/item?id=5660797
  - Skalman 13y ago
    HN has been available over HTTPS for longer than that.
happyshadows 13y ago
I wish Google would change their meta-tag from "origin" to "always". It's getting harder and harder to see which keywords bring traffic to your site these days.
kijin 13y ago
The article makes it sound like every webmaster is entitled to see Referer headers. Really? How does enabling referers "save the world"? A few weeks ago, I ran across a website (can't remember their domain) that refused to show me any content unless I enabled third-party cookies, and even contained a lengthy argument explaining why disabling third-party cookies hurts the web. IIRC the whole argument was 24K bullshit, written by people who feel entitled to keep making money with their outdated business models and who were obviously alarmed because modern browsers were doing sensible things. And now we're seeing a very similar argument, only this time it's about referers. Why do you think it matters whether I came to your site via Google, Reddit, HN or someone else's blog? What makes you think gives you the right to know that? Webmasters never had the right to know where your visitors were coming from, any more than the owner of a random gas station on the Interstate has the right to know which city his customers are driving from. If SSL is making Referer headers disappear, good riddance. We just closed a privacy hole, 99 more to go. Next in the TODO list: get rid of referers even when the referring website doesn't use SSL, because as the article correctly points out, we've got a bit of inconsistency there.
- arb99 13y ago
  I use referrers extensively (see how sites are doing traffic wise, traffic trades etc) so I am definitely glad they exist, but i am always surprised it was ever added to the spec. think it was added in 1.1 (so 1999). Without it I guess a LOT of sites nowadays would link out with urls like ?from=mysite.com so people knew where traffic was coming from. It is, like you said, basically a massive privacy hole. But a very handy one
  - kijin 13y ago
    I think search engines like Google should just standardize on a query string like your example. It always works, it's clearly opt-in, it's visible to the user, it's just as straightforward as a header (if not more) for web apps to process, and it's no more spoofable than a header anyway. If you think it's a bad idea to pollute URLs with such things, an alternative would be a URL fragment that is detected by JS.
tarkin2 13y ago
I get referrers from httpS://www.google.com quite often in my logs. And I'm running my blog on HTTP. How is this so?
- slig 13y ago
  Looks like someone didn't RTFA.
- ZoFreX 13y ago
  Read the article. This is explained about 80% of the way through.
leephillips 13y ago
You can selectively block referrer information if you are using a webkit browser (Chrome, Safari, etc.) by using http://lee-phillips.org/norefBookmarklet/ http://lee-phillips.org/norefBookmarklet/ I've found the information in my referrer logs quite interesting and useful, despite the Russian referrer spam, and am sorry to see it going away.
zapt02 13y ago
If you enable SSL for your site, will you start seeing referrers again? Mainly interested in the case of Google searches.
- j_s 13y ago
  No. You have to pay if you want referrer info from Google. http://searchengineland.com/google-puts-a-price-on-privacy-98029 http://searchengineland.com/google-puts-a-price-on-privacy-9...
  - tacticus 13y ago
    Or run https on your server.
    - j_s 13y ago
      The point is that Google obfuscates their search result links so that they do not include search keywords any more -- if you are interested in knowing the keywords, you [typically] have to pay. If you are just looking to know that the referrer was Google, then yes you can see that. However, this is not really useful information to most people. They implement this in two ways: (1) If you go directly to google.com and type in your search, the results page uses a # in the url which keeps all the query parameters out of the referrer. (2) Google has used (not sure if they still do/randomly test whether or not to) JavaScript redirects which overwrite the url when a search result is clicked. I'm sure there are other ways for Google to hide the referrer -- plus Google and various browser extensions can turn parts of this on/off however they choose. It is still possible to wind up with a referrer from a Google search where you can see the search keywords, if for example the search is done using the browser address/search bar, and the JavaScript overwriting result urls is not active (turned off by NoScript, etc.). However, this is not in Google's best business interest (if they can convince people to pay for the info) so I am counting on them trending towards making this the least likely of possible scenarios.
- jmillikin 13y ago
  Yes. I just tested it by searching for [haskell dbus] and clicking a result link to my site. Here's the referer in my nginx log: 2620:0:1000:3509:ed78:b06d:4942:90e1 - - [28/May/2013:17:35:00 +0000] "GET /software/haskell-dbus/ HTTP/1.1" 200 5819 "https://www.google.com/search?client=ubuntu&channel=fs&q=haskell+dbus&ie=utf-8&oe=utf-8" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:21.0) Gecko/20100101 Firefox/21.0"

110 points by Smerity 13y ago

I already hide/change my referrer through browser extensions, though I guess this might finally get those few sites who have functionality based on refs to stop using it.

Smerity 13y ago
Whilst I still think referrers should be used where appropriate, there are certainly places or reasons you want to nix them. HTTPS not passing along referrers by default is a sensible decision, as is the addition of <meta name="referrer" content="never"> for sites stuck in HTTP that may want to go under the radar. I still feel that removing referrers entirely destroys many useful tools and analytics that we've traditionally been able to use. It removes the core way in which we understand connections across the Internet. By removing referrers, the best we can do is use link graphs, falling back to the original PageRank algorithm where we assume people are random bots that click on one of the links on the page. Edit: Can't reply to you hnriot due to comment depth limit. My reference to PageRank is as links and backlinks could be used as a poor referrer substitute, though they aren't currently used as it's a lot more work and less accurate. I'm simply saying that, in the event that referrers all disappeared tomorrow, you'd see normal websites trying to estimate where their traffic comes from by using a PageRank inspired algorithm, or more naively by looking at who links where.
- hnriot 13y ago
  Referrers have nothing to do with pagerank. It's used in website analytics of course, and, in general is a very bad idea leaking information between websites. Search engines don't see your site's referrer data so the association you tried to create with pagerank is misleading.
  - encoderer 13y ago
    I, for one, didn't think his argument was remotely misleading. I did, however, think that bringing PageRank into the argument was ill advised and probably detrimental to his goal of advocacy through education. It just confused things, introduced another rabbit-hole concept. If the author thought that mentioning PageRank would lend credibility to the 'link counting' / 'random link clicking bots' foregone conclusion he setup., he was right. It did. So link the text to a footnote referencing PageRank and stay on topic.

anonymouz 13y ago

Referers are a privacy trade-off though, and personally I have never seen a good reason why the desire of some website to track where I am coming from should trump my desire to not broadcast this information to the world. Therefore I like to switch of referers anyway.

claudius 13y ago
I am doing the same. Fortunately, the websites requiring the referer to be set are getting fewer and fewer – some years ago, I regularly got placeholder images ‘THIS IMAGE WAS STOLEN FROM XYZ’ when browsing XYZ without referers.
- joosters 13y ago
  This is one of the good uses of HTTP referrers! I host a small, very low traffic website. One day, the bandwidth shoots through the roof and stays high. The reason? One of the images on a page got added to the .sig of someone in a popular forum. Suddenly thousands of people are fetching the image. The solution was to filter by referrer header, letting the image be seen by visitors to the actual page, but linking from other sites gets blocked. Note that usually it's best to allow requests that have no referrer header at all, otherwise you'll be blocking some legitimate viewers of your site. End result: bandwidth back down to the usual, tiny levels. It's not that I cared about people copying the images, I just didn't want to foot the bill for the traffic!
  - graue 13y ago
    I think you misunderstand — this seems to have been claudius' point: > Note that usually it's best to allow requests that have no referrer header at all, otherwise you'll be blocking some legitimate viewers of your site.
    - joosters 13y ago
      It was probably a badly-placed reply; my point was to illustrate a use for referrer headers and to state that sites blocking based on the lack of a referrer were doing it wrong.

lucb1e 13y ago

Oh I did not know this. Very good to know indeed! I too am missing referrers from websites, and since my website also supports https I suppose others are missing them from me as well. This way I can make the internet a very tiny bit better ^^.

mooism2 13y ago

So my understanding is that the default behaviour is: 1. Follow link from https://example.org https://example.org to http://example.com http://example.com --- referrer is not sent 2. Follow link from https://example.org https://example.org to https://example.com https://example.com --- referrer is sent I don't understand how the same referrer can be too sensitive to be sent as plaintext, but harmless enough to be passed to a not-necessarily-trusted third party.

deleted 13y ago
[deleted]
kyrias 13y ago
It's more about who can see it along the way than it is about who receives the request.
- ctz 13y ago
  I'm willing to bet that it gets sent even if the second HTTPS connection provides no effective security. (If this is the protocol we're building economies on these days, I feel computer security is going to get much worse before it gets better...)
  - joosters 13y ago
    If your browser is communicating on the second HTTPS connection, that pretty much implies that you are happy with its security levels, right? After all, if you weren't (e.g. bad certificates, etc), your browser wouldn't have established & confirmed the secure connection, HTTP referrer headers or not.
    - ctz 13y ago
      You're right, of course. Let me expand my argument a little. We have two security domains, associated with the first and second server. The first server has sensitive URLs, the second receives these in the referrer header. In making the first connection, the client and first server get to make some security policy decisions: the crypto used in securing the connection, the availability of countermeasures against known TLS protocol vulnerabilities, client and server authentication methods, etc. These parameters are complex, and you end up with a connection with some security level given some attacker model. If you think about it really hard, you can come up with a quantitive estimate of what security you might get from the connection in terms of attacker work -- anywhere from zero (trivially broken) to 256-bit security (very good). Assume our first connection has a 128-bit security level and has countermeasures for TLS vulnerabilities. This is good going. Now we serve over this connection a https link, which the user clicks. Unfortunately the client and/or second server is poorly configured and we only manage (for the sake of argument) a 40-bit security level and no TLS vulnerability countermeasures. Now we've reduced portions of the first connection to the security level offerred by the second connection. This is really very surprising.

dlsym 13y ago

But why is that a bad thing? Just count your page impressions to see if you blogcontent is popular. What websites your visitors consume is IMHO not your business.

josteink 13y ago
> What websites your visitors consume is IMHO not your business. But if your product is the subject of discussion on another page which links to yours, you may want to be able to join the discussion. As a site owner, I don't think it's unreasonable to ask my visitors how and where they found out about my website.
jonknee 13y ago
It's two very different things if the sudden spike in page views is because of nytimes.com or reddit.com.

w00kie 13y ago

Thanks for the info, I was not aware of this meta tag. My blog is all HTTPS (with SPDY) and I added content="origin" to all visitor facing pages, leaving the admin pages dark.

cnahr 13y ago

When I recently tried to purge my server logs of referrer spam, I found that a whopping 80% of all visitors had either no referrer or referred from my own website. About a third of the rest were spam. So judging from my small sample, referrer headers are largely useless except as spam machines. They should probably be removed from HTTP anyway.

kmasters 13y ago

I fail to see why referrer is part of any spec. Referrer is a 90's concoction when we were all so naive of future uses. I'm sure it's made a lot of porn sites some money, but affiliate marketing is not the purpose of web standards. As far as analytics its all done with tracking cookies now, which is another issue that needs to be addressed, but Im not going to miss referer should it ever really go away.

jacques_chester 13y ago
My honours project was on the problems of tracking users visiting different websites using cookies. The problems are numerous. I think I went through 9 revisions of the "naive protocol" (including, at one point, ditching the referer header in favour of HTTPS). Subsequently I realised that my tracking protocol was broken anyhow. My conclusion is that there's no reliable way to track users visiting multiple websites using the standard features of HTML/JS/HTTP in the face of malicious users or publishers. You have to fall back on traffic analysis. I developed a successor technology which works better in many respects (but not all). As it's the subject of a current patent application I can't really go into much detail.
- mayank 13y ago
  I hope you've looked at evercookie before you go about filing patents on user tracking...
  - jacques_chester 13y ago
    It's a quite different scheme.
Angostura 13y ago
I see it rather differently. Referrer is a 1990s concoction based on the idea that the Web is a collaborative effort that we all engage in. I don't personally have any problem in a Web site knowing where I arrive from in most cases. It's harmless him of information (in most cases, as I say) which makes the Web site owner's life a little better.

kvi 13y ago

Yeah, that's a long known thing. I use HTTPS on parsebin.com specifically for stripping out referrers.

nathell 13y ago

This is how I discovered HN went HTTPS some time ago. I never noticed until now.

sp332 13y ago
HN started using SPDY recently, and SPDY requires HTTPS. https://news.ycombinator.com/item?id=5660797 https://news.ycombinator.com/item?id=5660797
- Skalman 13y ago
  HN has been available over HTTPS for longer than that.

happyshadows 13y ago

I wish Google would change their meta-tag from "origin" to "always". It's getting harder and harder to see which keywords bring traffic to your site these days.

kijin 13y ago

The article makes it sound like every webmaster is entitled to see Referer headers. Really? How does enabling referers "save the world"? A few weeks ago, I ran across a website (can't remember their domain) that refused to show me any content unless I enabled third-party cookies, and even contained a lengthy argument explaining why disabling third-party cookies hurts the web. IIRC the whole argument was 24K bullshit, written by people who feel entitled to keep making money with their outdated business models and who were obviously alarmed because modern browsers were doing sensible things. And now we're seeing a very similar argument, only this time it's about referers. Why do you think it matters whether I came to your site via Google, Reddit, HN or someone else's blog? What makes you think gives you the right to know that? Webmasters never had the right to know where your visitors were coming from, any more than the owner of a random gas station on the Interstate has the right to know which city his customers are driving from. If SSL is making Referer headers disappear, good riddance. We just closed a privacy hole, 99 more to go. Next in the TODO list: get rid of referers even when the referring website doesn't use SSL, because as the article correctly points out, we've got a bit of inconsistency there.

arb99 13y ago
I use referrers extensively (see how sites are doing traffic wise, traffic trades etc) so I am definitely glad they exist, but i am always surprised it was ever added to the spec. think it was added in 1.1 (so 1999). Without it I guess a LOT of sites nowadays would link out with urls like ?from=mysite.com so people knew where traffic was coming from. It is, like you said, basically a massive privacy hole. But a very handy one
- kijin 13y ago
  I think search engines like Google should just standardize on a query string like your example. It always works, it's clearly opt-in, it's visible to the user, it's just as straightforward as a header (if not more) for web apps to process, and it's no more spoofable than a header anyway. If you think it's a bad idea to pollute URLs with such things, an alternative would be a URL fragment that is detected by JS.

tarkin2 13y ago

I get referrers from httpS://www.google.com quite often in my logs. And I'm running my blog on HTTP. How is this so?

slig 13y ago
Looks like someone didn't RTFA.
ZoFreX 13y ago
Read the article. This is explained about 80% of the way through.

leephillips 13y ago

You can selectively block referrer information if you are using a webkit browser (Chrome, Safari, etc.) by using http://lee-phillips.org/norefBookmarklet/ http://lee-phillips.org/norefBookmarklet/ I've found the information in my referrer logs quite interesting and useful, despite the Russian referrer spam, and am sorry to see it going away.

zapt02 13y ago

If you enable SSL for your site, will you start seeing referrers again? Mainly interested in the case of Google searches.

j_s 13y ago
No. You have to pay if you want referrer info from Google. http://searchengineland.com/google-puts-a-price-on-privacy-98029 http://searchengineland.com/google-puts-a-price-on-privacy-9...
- tacticus 13y ago
  Or run https on your server.
  - j_s 13y ago
    The point is that Google obfuscates their search result links so that they do not include search keywords any more -- if you are interested in knowing the keywords, you [typically] have to pay. If you are just looking to know that the referrer was Google, then yes you can see that. However, this is not really useful information to most people. They implement this in two ways: (1) If you go directly to google.com and type in your search, the results page uses a # in the url which keeps all the query parameters out of the referrer. (2) Google has used (not sure if they still do/randomly test whether or not to) JavaScript redirects which overwrite the url when a search result is clicked. I'm sure there are other ways for Google to hide the referrer -- plus Google and various browser extensions can turn parts of this on/off however they choose. It is still possible to wind up with a referrer from a Google search where you can see the search keywords, if for example the search is done using the browser address/search bar, and the JavaScript overwriting result urls is not active (turned off by NoScript, etc.). However, this is not in Google's best business interest (if they can convince people to pay for the info) so I am counting on them trending towards making this the least likely of possible scenarios.
jmillikin 13y ago
Yes. I just tested it by searching for [haskell dbus] and clicking a result link to my site. Here's the referer in my nginx log: 2620:0:1000:3509:ed78:b06d:4942:90e1 - - [28/May/2013:17:35:00 +0000] "GET /software/haskell-dbus/ HTTP/1.1" 200 5819 "https://www.google.com/search?client=ubuntu&channel=fs&q=haskell+dbus&ie=utf-8&oe=utf-8" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:21.0) Gecko/20100101 Firefox/21.0"