10 ms·

Where did all the HTTP referrers go? (2013)

97 points by bemmu 11y ago

ComputerGuru 11y ago
Google originally justified their blocking of search term data in HTTP referrer fields as being due to the security implications of HTTPS vs HTTP. The fact that they now use <meta name="referrer" content="origin"> proves it was just an excuse to obscure that data, since they have it at their disposal to transmit that information securely to the destination site.
- byuu 11y ago
  All of this has always been about forcing everyone onto HTTPS at any cost. Denying referral headers, not allowing HTTP/2 over HTTP, treating self-signed certificates as the worst thing ever, soon putting red X'es in the URL bars, etc. It's all disgustingly heavy-handed. But they get away with it because most people are on board with it. They seem to lack empathy of how this would look if they weren't on board with the change. The general idea is a great one: who doesn't love extra security? But all of those cons ... interfering with caching, costing money for wildcard certificates, all the security vulnerabilities in the TLS libraries, the (miniscule, but non-zero) extra computational power required, the added setup difficulty (yes, even if you trust Let's Encrypt to execute code on your box, it's still an added burden), putting trust in the CA system that has shown several major flaws already (such as rogue certificates) ... all conveniently ignored or whitewashed away. But all the posturing in the world won't be enough to eliminate HTTP from the web. It's going to outlive all of us. Not even because people like me want it to, but because there's just so much legacy code out there that's never going to get touched. Millions of devices and applications that only speak HTTP, that nobody's ever going to update. Hell, it's trivial to find Gopher proxy layers, and that never even got 1% of the uptake that HTTP has today.
  - otabdeveloper 11y ago
    This would have been OK if HTTPS truly did increase security. In reality, however, the sensitive part is the 'who' and 'when', not the 'what'. HTTPS makes sense for hiding passwords or bank data, for everything else it's mostly pointless. (If you've watched any police procedural shows you've noticed that they don't need wiretaps to make an arrest, a destination phone number and time is all that's needed.)
    - seabee 11y ago
      Have we already forgotten FireSheep?
  - Ono-Sendai 11y ago
    I agree the whitewashing is annoying. One correction: the 'miniscule, but non-zero' computational power for TLS connections, in particular the handshake, is actually quite significant. (like 0.4s of CPU time in my tests). So the whitewashing problem may be worse than you think!
- jakobegger 11y ago
  As a user, I'm glad they are hiding keywords. First of all it is private data, and secondly websites where doing all kinds of annoying stuff with the information (eg. they added a list of search terms used to find the page on the bottom of the page, presumably to further increase Google rank for these terms). But I really miss the time when I saw the keywords that people used to find my page. Lots of insight that is now lost.
jstayton 11y ago
Current browser support: http://caniuse.com/#feat=referrer-policy http://caniuse.com/#feat=referrer-policy
Houshalter 11y ago
I use referrer block for Chrome. I have only twice had anything break from it, and it was a simple fix to whitelist that website. Disqus and coursera both break without referrer. I don't know why everyone doesn't block referrer. It seems like such a massive breach of privacy to leak that information on every link you click.
- arm 11y ago
  In Safari (the OS X version), you can also do this with JS Blocker: http://jsblocker.toggleable.com/ http://jsblocker.toggleable.com/
- atriix 11y ago
  Since you mentioned whitelistnig I assume an extension, care share which one?
- hackuser 11y ago
  Referrer Control in Firefox works very well: https://addons.mozilla.org/EN-US/firefox/addon/referrer-control/ https://addons.mozilla.org/EN-US/firefox/addon/referrer-cont...
- leephillips 11y ago
  Referrers are useful and can be a friendly, automatic way of letting someone know how you found their page. I've discovered many links to my pages on other websites, and whole articles discussing my articles, through referrers. Since none of the other pinging mechanisms ever gained universal traction, it's the best thing we have, and I'm sorry that it's all but disappeared. And I remember how amazed I was to discover the existence of "referrer spam" years ago. You can block referrers in a number of ways whenever you'd like, for example with a simple bookmarklet: http://lee-phillips.org/norefBookmarklet/ http://lee-phillips.org/norefBookmarklet/ Finally, looking at search terms that led to your site can be very entertaining.
davecardwell 11y ago
Note: Authors are encouraged to avoid the legacy keywords never, default, and always. The keywords none, none-when-downgrade, and unsafe-url respectively are preferred. - https://www.w3.org/TR/referrer-policy/#referrer-policy-delivery-meta https://www.w3.org/TR/referrer-policy/#referrer-policy-deliv... Although the current versions of Edge only support the legacy keywords according to http://caniuse.com/#feat=referrer-policy http://caniuse.com/#feat=referrer-policy
- tempestn 11y ago
  Do you happen to know what Edge defaults to if it doesn't recognize the value? One would hope to "never"?
  - davecardwell 11y ago
    I don’t know, sorry, although I would suspect they stick to “default”.
    - tempestn 11y ago
      Just tested this myself and it looks like you're right. With meta referrer set to content=never, Edge passes the referrer. With content=none it doesn't. Chrome is the opposite, and Firefox does not send the referrer in either case. (And of course IE sends it in both cases, since it doesn't recognize the referrer meta tag.)
  - deleted 11y ago
    [deleted]
- mike-cardwell 11y ago
  I've been using "no-referrer" rather than "none" as per https://w3c.github.io/webappsec-referrer-policy/ https://w3c.github.io/webappsec-referrer-policy/ which is linked to from the "caniuse.com" link you quoted above. Am I wrong?
  - davecardwell 11y ago
Smerity 11y ago
As the author, this was a pleasant surprise! :) It was written in 2013 (potentially note that in the title?) though people still frequently say they find it useful. As @jstayton noted, browser support has come a long way[1], and @davecardwell notes that some of the keywords are deprecated[2]. I really should update both browser support and the "which websites use it" part. Hacker News now supports it for example, which I'm hugely glad to see! ^_^ Previous HN discussion: https://news.ycombinator.com/item?id=5778444 https://news.ycombinator.com/item?id=5778444 [1]: http://caniuse.com/#feat=referrer-policy http://caniuse.com/#feat=referrer-policy [2]: https://www.w3.org/TR/referrer-policy/#referrer-policy-states https://www.w3.org/TR/referrer-policy/#referrer-policy-state...
- pbreit 11y ago
  I didn't see this explicitly mentioned but if my site is HTTPS, I get referers, right? And so isn't that the easiest/best solution?
  - Smerity 11y ago
    From the article: > HTTPS websites will send referrers to any other HTTPS website even if it contains sensitive information As such, having a website which is HTTPS will get you referers from both HTTP and HTTPS if they're willing to send them, but it's also important to know you can control them in case there are privacy implications such as leaking your customer's information to external HTTPS sites.
    - sopooneo 11y ago
      Thank you for this, because I think I'm about to learn from you, but I'm confused. How can you control them? Is there some way for a site to signal to a browser whether it should or send referrer headers and under what circumstances?
      - deleted 11y ago
        [deleted]
      - 11y ago
deleted 11y ago
[deleted]
tempestn 11y ago
Does anyone know the preferred (current) method of sending referrer directives between the referrer-policy header[1] and the referrer directive under Content Security Policy (CSP) 1.1[2]? Since these specs are evolving, there is a lot of contradictory documentation online, and it's tough to weed out what's the accepted solution (if there is one). Presumably using headers (of some sort) is preferable to meta tags where possible though? Edit: As mentioned by davecardwell, the always/never/default settings, which are referenced in my CSP link there, are deprecated. Perhaps the whole concept of serving the referrer policy via CSP is as well? [1] https://w3c.github.io/webappsec-referrer-policy/#referrer-policy-delivery https://w3c.github.io/webappsec-referrer-policy/#referrer-po... [2] https://www.w3.org/TR/2014/WD-CSP11-20140211/#referrer https://www.w3.org/TR/2014/WD-CSP11-20140211/#referrer
IgorPartola 11y ago
Don't you mean referer? Edit: yes, I know. This was an HTTP joke. Not a very funny one. Here is another one: what is the difference between a hippo and a zippo? One is very heavy, the other is a little lighter.
- xenophonf 11y ago
  We're probably better off spelling "referrer" correctly except in packet traces or HTTP protocol implementations.
- Buge 11y ago
  That's covered in the appendix.
- malsun 11y ago
  Referrer is the correct English spelling. It's so easy to accidentally create a bug by forgetting that the standard relies on a spelling mistake.
EGreg 11y ago
Or you could just, you know, not include sensitive information in URLs, reducing the risk of it being leaked through sharing, referer, or other ways. HTTPS only prevents MITM. You should still think about the design if your site.
merb 11y ago
Actually I like the fact that some people won't add a referrer meta. Not everbody needs to know where I came from.
obelisk_ 11y ago
I knew about what happened with the referrers when going from HTTP to HTTPS but had not heard about the meta referrer and have been wondering why, even though my own site is HTTPS I see so many bare domain referrers. Now I know. Thank you both to author and submitter.

97 points by bemmu 11y ago

Google originally justified their blocking of search term data in HTTP referrer fields as being due to the security implications of HTTPS vs HTTP. The fact that they now use <meta name="referrer" content="origin"> proves it was just an excuse to obscure that data, since they have it at their disposal to transmit that information securely to the destination site.

byuu 11y ago
All of this has always been about forcing everyone onto HTTPS at any cost. Denying referral headers, not allowing HTTP/2 over HTTP, treating self-signed certificates as the worst thing ever, soon putting red X'es in the URL bars, etc. It's all disgustingly heavy-handed. But they get away with it because most people are on board with it. They seem to lack empathy of how this would look if they weren't on board with the change. The general idea is a great one: who doesn't love extra security? But all of those cons ... interfering with caching, costing money for wildcard certificates, all the security vulnerabilities in the TLS libraries, the (miniscule, but non-zero) extra computational power required, the added setup difficulty (yes, even if you trust Let's Encrypt to execute code on your box, it's still an added burden), putting trust in the CA system that has shown several major flaws already (such as rogue certificates) ... all conveniently ignored or whitewashed away. But all the posturing in the world won't be enough to eliminate HTTP from the web. It's going to outlive all of us. Not even because people like me want it to, but because there's just so much legacy code out there that's never going to get touched. Millions of devices and applications that only speak HTTP, that nobody's ever going to update. Hell, it's trivial to find Gopher proxy layers, and that never even got 1% of the uptake that HTTP has today.
- otabdeveloper 11y ago
  This would have been OK if HTTPS truly did increase security. In reality, however, the sensitive part is the 'who' and 'when', not the 'what'. HTTPS makes sense for hiding passwords or bank data, for everything else it's mostly pointless. (If you've watched any police procedural shows you've noticed that they don't need wiretaps to make an arrest, a destination phone number and time is all that's needed.)
  - seabee 11y ago
    Have we already forgotten FireSheep?
- Ono-Sendai 11y ago
  I agree the whitewashing is annoying. One correction: the 'miniscule, but non-zero' computational power for TLS connections, in particular the handshake, is actually quite significant. (like 0.4s of CPU time in my tests). So the whitewashing problem may be worse than you think!
jakobegger 11y ago
As a user, I'm glad they are hiding keywords. First of all it is private data, and secondly websites where doing all kinds of annoying stuff with the information (eg. they added a list of search terms used to find the page on the bottom of the page, presumably to further increase Google rank for these terms). But I really miss the time when I saw the keywords that people used to find my page. Lots of insight that is now lost.

jstayton 11y ago

Current browser support: http://caniuse.com/#feat=referrer-policy http://caniuse.com/#feat=referrer-policy

Houshalter 11y ago

I use referrer block for Chrome. I have only twice had anything break from it, and it was a simple fix to whitelist that website. Disqus and coursera both break without referrer. I don't know why everyone doesn't block referrer. It seems like such a massive breach of privacy to leak that information on every link you click.

arm 11y ago
In Safari (the OS X version), you can also do this with JS Blocker: http://jsblocker.toggleable.com/ http://jsblocker.toggleable.com/
atriix 11y ago
Since you mentioned whitelistnig I assume an extension, care share which one?
hackuser 11y ago
Referrer Control in Firefox works very well: https://addons.mozilla.org/EN-US/firefox/addon/referrer-control/ https://addons.mozilla.org/EN-US/firefox/addon/referrer-cont...
leephillips 11y ago
Referrers are useful and can be a friendly, automatic way of letting someone know how you found their page. I've discovered many links to my pages on other websites, and whole articles discussing my articles, through referrers. Since none of the other pinging mechanisms ever gained universal traction, it's the best thing we have, and I'm sorry that it's all but disappeared. And I remember how amazed I was to discover the existence of "referrer spam" years ago. You can block referrers in a number of ways whenever you'd like, for example with a simple bookmarklet: http://lee-phillips.org/norefBookmarklet/ http://lee-phillips.org/norefBookmarklet/ Finally, looking at search terms that led to your site can be very entertaining.

davecardwell 11y ago

Note: Authors are encouraged to avoid the legacy keywords never, default, and always. The keywords none, none-when-downgrade, and unsafe-url respectively are preferred. - https://www.w3.org/TR/referrer-policy/#referrer-policy-delivery-meta https://www.w3.org/TR/referrer-policy/#referrer-policy-deliv... Although the current versions of Edge only support the legacy keywords according to http://caniuse.com/#feat=referrer-policy http://caniuse.com/#feat=referrer-policy

tempestn 11y ago
Do you happen to know what Edge defaults to if it doesn't recognize the value? One would hope to "never"?
- davecardwell 11y ago
  I don’t know, sorry, although I would suspect they stick to “default”.
  - tempestn 11y ago
    Just tested this myself and it looks like you're right. With meta referrer set to content=never, Edge passes the referrer. With content=none it doesn't. Chrome is the opposite, and Firefox does not send the referrer in either case. (And of course IE sends it in both cases, since it doesn't recognize the referrer meta tag.)
- deleted 11y ago
  [deleted]
mike-cardwell 11y ago
I've been using "no-referrer" rather than "none" as per https://w3c.github.io/webappsec-referrer-policy/ https://w3c.github.io/webappsec-referrer-policy/ which is linked to from the "caniuse.com" link you quoted above. Am I wrong?
- davecardwell 11y ago

Smerity 11y ago

As the author, this was a pleasant surprise! :) It was written in 2013 (potentially note that in the title?) though people still frequently say they find it useful. As @jstayton noted, browser support has come a long way[1], and @davecardwell notes that some of the keywords are deprecated[2]. I really should update both browser support and the "which websites use it" part. Hacker News now supports it for example, which I'm hugely glad to see! ^_^ Previous HN discussion: https://news.ycombinator.com/item?id=5778444 https://news.ycombinator.com/item?id=5778444 [1]: http://caniuse.com/#feat=referrer-policy http://caniuse.com/#feat=referrer-policy [2]: https://www.w3.org/TR/referrer-policy/#referrer-policy-states https://www.w3.org/TR/referrer-policy/#referrer-policy-state...

pbreit 11y ago
I didn't see this explicitly mentioned but if my site is HTTPS, I get referers, right? And so isn't that the easiest/best solution?
- Smerity 11y ago
  From the article: > HTTPS websites will send referrers to any other HTTPS website even if it contains sensitive information As such, having a website which is HTTPS will get you referers from both HTTP and HTTPS if they're willing to send them, but it's also important to know you can control them in case there are privacy implications such as leaking your customer's information to external HTTPS sites.
  - sopooneo 11y ago
    Thank you for this, because I think I'm about to learn from you, but I'm confused. How can you control them? Is there some way for a site to signal to a browser whether it should or send referrer headers and under what circumstances?
    - deleted 11y ago
      [deleted]
    - 11y ago

deleted 11y ago

[deleted]

tempestn 11y ago

Does anyone know the preferred (current) method of sending referrer directives between the referrer-policy header[1] and the referrer directive under Content Security Policy (CSP) 1.1[2]? Since these specs are evolving, there is a lot of contradictory documentation online, and it's tough to weed out what's the accepted solution (if there is one). Presumably using headers (of some sort) is preferable to meta tags where possible though? Edit: As mentioned by davecardwell, the always/never/default settings, which are referenced in my CSP link there, are deprecated. Perhaps the whole concept of serving the referrer policy via CSP is as well? [1] https://w3c.github.io/webappsec-referrer-policy/#referrer-policy-delivery https://w3c.github.io/webappsec-referrer-policy/#referrer-po... [2] https://www.w3.org/TR/2014/WD-CSP11-20140211/#referrer https://www.w3.org/TR/2014/WD-CSP11-20140211/#referrer

IgorPartola 11y ago

Don't you mean referer? Edit: yes, I know. This was an HTTP joke. Not a very funny one. Here is another one: what is the difference between a hippo and a zippo? One is very heavy, the other is a little lighter.

xenophonf 11y ago
We're probably better off spelling "referrer" correctly except in packet traces or HTTP protocol implementations.
Buge 11y ago
That's covered in the appendix.
malsun 11y ago
Referrer is the correct English spelling. It's so easy to accidentally create a bug by forgetting that the standard relies on a spelling mistake.

EGreg 11y ago

Or you could just, you know, not include sensitive information in URLs, reducing the risk of it being leaked through sharing, referer, or other ways. HTTPS only prevents MITM. You should still think about the design if your site.

merb 11y ago

Actually I like the fact that some people won't add a referrer meta. Not everbody needs to know where I came from.

obelisk_ 11y ago

I knew about what happened with the referrers when going from HTTP to HTTPS but had not heard about the meta referrer and have been wondering why, even though my own site is HTTPS I see so many bare domain referrers. Now I know. Thank you both to author and submitter.