6 ms·

The article makes it sound like every webmaster is entitled to see Referer headers. Really? How does enabling referers "save the world"? A few weeks ago, I ran

by kijin 13y ago

The article makes it sound like every webmaster is entitled to see Referer headers. Really? How does enabling referers "save the world"? A few weeks ago, I ran across a website (can't remember their domain) that refused to show me any content unless I enabled third-party cookies, and even contained a lengthy argument explaining why disabling third-party cookies hurts the web. IIRC the whole argument was 24K bullshit, written by people who feel entitled to keep making money with their outdated business models and who were obviously alarmed because modern browsers were doing sensible things. And now we're seeing a very similar argument, only this time it's about referers. Why do you think it matters whether I came to your site via Google, Reddit, HN or someone else's blog? What makes you think gives you the right to know that? Webmasters never had the right to know where your visitors were coming from, any more than the owner of a random gas station on the Interstate has the right to know which city his customers are driving from. If SSL is making Referer headers disappear, good riddance. We just closed a privacy hole, 99 more to go. Next in the TODO list: get rid of referers even when the referring website doesn't use SSL, because as the article correctly points out, we've got a bit of inconsistency there.

arb99 13y ago
I use referrers extensively (see how sites are doing traffic wise, traffic trades etc) so I am definitely glad they exist, but i am always surprised it was ever added to the spec. think it was added in 1.1 (so 1999). Without it I guess a LOT of sites nowadays would link out with urls like ?from=mysite.com so people knew where traffic was coming from. It is, like you said, basically a massive privacy hole. But a very handy one
- kijin 13y ago
  I think search engines like Google should just standardize on a query string like your example. It always works, it's clearly opt-in, it's visible to the user, it's just as straightforward as a header (if not more) for web apps to process, and it's no more spoofable than a header anyway. If you think it's a bad idea to pollute URLs with such things, an alternative would be a URL fragment that is detected by JS.
  - toast0 13y ago
    Query string wouldn't always work, because the site may already use the query argument for something else; some sites may break or freak out when given unexpected arguments. URL fragments aren't transmitted to the server, so that means you lose referrer analysis for static sites.
pi18n 13y ago
And there are occasionally similar articles where the author informs us all that AdBlock and NoScript are literally destroying the internet. And that misguided blogger this week who wants Apple to force users to accept push notifications from any app they've installed to make his life easier.
Smerity 13y ago
If you reread the article, I actually say you should add a meta header regardless of whether you do or don't want to send headers. Why? It actually solves your "next in the TODO list". Most web sites that shouldn't send referrers don't use <meta name="referrer" content="never"> so will be leaking referrers to other web sites. Adding this meta tag will eliminate referrers in both HTTP and HTTPS. So yes, my own preference is to keep HTTP Referrers, but I also explain how to kill HTTP referrers for webmasters who would like to as well.
- untog 13y ago
  That's still taking the decision out of the hands of the user, though. I'm not saying that I totally agree with kijin, but I don't think your answer addresses his point.
  - Smerity 13y ago
    I didn't realise I missed a point to address, I'll seek to clarify. It's always the user's choice as to whether to send referrers or not, as the referrer is actually added by the user's web browser itself. Extensions exist for just about every major web browser[1][2][...] to modify the behaviour of the HTTP Referrer field. If you don't like the idea of sending referrers, it's entirely within your control to never send a single referrer. In almost all cases, disabling the referrer entirely won't result in any broken behaviour, primarily as the HTTP Referrer is unreliable and can be spoofed anyway. [1]: https://chrome.google.com/webstore/detail/referer-control/hnkcfpcejkafcihlgbojoidoihckciin?hl=en https://chrome.google.com/webstore/detail/referer-control/hn... [2]: https://addons.mozilla.org/en-US/firefox/addon/refcontrol/ https://addons.mozilla.org/en-US/firefox/addon/refcontrol/
- kijin 13y ago
  I disagree with your suggestion to use such a meta tag, because it's an opt-out scheme. People should not have to opt out of potential privacy leaks. If a web page does not specify any referer policy, I think the default should be no referer. Unfortunately I still can't remember the website where I found the "bullshit" argument, and it's not in my history because I probably ended up using a different browser to comply with their no-access-unless-you-accept-3rd-party-cookies policy. But if you understand why people like me want third-party cookies to be disabled by default, I think you'll also understand why I want referers to be disabled by default, too. It's not about user control as @untog suggests, because the user is always ultimately in control when it comes to HTTP headers. Rather, it's about having secure defaults.
untog 13y ago
There are some legitimate (IMO) uses of referrers, though. Preventing hot-linking, for example.
- vidarh 13y ago
  There's many legitimate uses for them, but while they're convenient for preventing hot linking, they're not necessary. Preventing hotlinking can be done easily enough by appending a url argument that changes regularly, and can be made performant by adding that argument checking to the configuration of your frontend/caching servers.
  - laumars 13y ago
    Not a bad idea, but you could probably just use the session ID (or a unique ID linked to the users session if it's a secure site). So when the session expires, so does access to the content. The drawback to this is you're exponentially increasing DB i/o. Personally though, I'd rather not block hotlinking. But I understand why some people are against it.
    - devicenull 13y ago
      It's actually implemented in a better way (no db required) with most web servers: http://wiki.nginx.org/HttpSecureLinkModule http://wiki.nginx.org/HttpSecureLinkModule
- tomjen3 13y ago
  No. The web is about links, if you don't want things to be linked too, don't make them accessible to the general public.
  - krapp 13y ago
    You can responsibly link to an image on someone else's server without embedding it directly on your site. The web is no longer just hypertext, and the potential inequity in resources and scalability between servers matters. Why should I be expected to suffer and pay for anyone who decides they like a particular image on my site and want to use it in their forum avi? They can just as easily copy it locally and use it.
liotier 13y ago
> The article makes it sound like every webmaster is entitled to see Referer headers. [..] Why do you think it matters whether I came to your site via Google, Reddit, HN or someone else's blog ? As an occasional small-time blogger, I like to know where the conversation happens, where people are interested - it enhances my engagement with my readers. As a reader, I'll gladly grant webmasters that courtesy.
pbreit 13y ago
Uh...I don't think anyone, OP included, is suggesting it's a right. Just that it used to be almost always available. And it's going away, sort of accidentally. On the scale of "useful things" vs "privacy invasions", it's pretty safely on the former side.

by kijin 13y ago

arb99 13y ago

I use referrers extensively (see how sites are doing traffic wise, traffic trades etc) so I am definitely glad they exist, but i am always surprised it was ever added to the spec. think it was added in 1.1 (so 1999). Without it I guess a LOT of sites nowadays would link out with urls like ?from=mysite.com so people knew where traffic was coming from. It is, like you said, basically a massive privacy hole. But a very handy one

kijin 13y ago
I think search engines like Google should just standardize on a query string like your example. It always works, it's clearly opt-in, it's visible to the user, it's just as straightforward as a header (if not more) for web apps to process, and it's no more spoofable than a header anyway. If you think it's a bad idea to pollute URLs with such things, an alternative would be a URL fragment that is detected by JS.
- toast0 13y ago
  Query string wouldn't always work, because the site may already use the query argument for something else; some sites may break or freak out when given unexpected arguments. URL fragments aren't transmitted to the server, so that means you lose referrer analysis for static sites.

pi18n 13y ago

And there are occasionally similar articles where the author informs us all that AdBlock and NoScript are literally destroying the internet. And that misguided blogger this week who wants Apple to force users to accept push notifications from any app they've installed to make his life easier.

Smerity 13y ago

If you reread the article, I actually say you should add a meta header regardless of whether you do or don't want to send headers. Why? It actually solves your "next in the TODO list". Most web sites that shouldn't send referrers don't use <meta name="referrer" content="never"> so will be leaking referrers to other web sites. Adding this meta tag will eliminate referrers in both HTTP and HTTPS. So yes, my own preference is to keep HTTP Referrers, but I also explain how to kill HTTP referrers for webmasters who would like to as well.

untog 13y ago
That's still taking the decision out of the hands of the user, though. I'm not saying that I totally agree with kijin, but I don't think your answer addresses his point.
- Smerity 13y ago
  I didn't realise I missed a point to address, I'll seek to clarify. It's always the user's choice as to whether to send referrers or not, as the referrer is actually added by the user's web browser itself. Extensions exist for just about every major web browser[1][2][...] to modify the behaviour of the HTTP Referrer field. If you don't like the idea of sending referrers, it's entirely within your control to never send a single referrer. In almost all cases, disabling the referrer entirely won't result in any broken behaviour, primarily as the HTTP Referrer is unreliable and can be spoofed anyway. [1]: https://chrome.google.com/webstore/detail/referer-control/hnkcfpcejkafcihlgbojoidoihckciin?hl=en https://chrome.google.com/webstore/detail/referer-control/hn... [2]: https://addons.mozilla.org/en-US/firefox/addon/refcontrol/ https://addons.mozilla.org/en-US/firefox/addon/refcontrol/
kijin 13y ago
I disagree with your suggestion to use such a meta tag, because it's an opt-out scheme. People should not have to opt out of potential privacy leaks. If a web page does not specify any referer policy, I think the default should be no referer. Unfortunately I still can't remember the website where I found the "bullshit" argument, and it's not in my history because I probably ended up using a different browser to comply with their no-access-unless-you-accept-3rd-party-cookies policy. But if you understand why people like me want third-party cookies to be disabled by default, I think you'll also understand why I want referers to be disabled by default, too. It's not about user control as @untog suggests, because the user is always ultimately in control when it comes to HTTP headers. Rather, it's about having secure defaults.

untog 13y ago

There are some legitimate (IMO) uses of referrers, though. Preventing hot-linking, for example.

vidarh 13y ago
There's many legitimate uses for them, but while they're convenient for preventing hot linking, they're not necessary. Preventing hotlinking can be done easily enough by appending a url argument that changes regularly, and can be made performant by adding that argument checking to the configuration of your frontend/caching servers.
- laumars 13y ago
  Not a bad idea, but you could probably just use the session ID (or a unique ID linked to the users session if it's a secure site). So when the session expires, so does access to the content. The drawback to this is you're exponentially increasing DB i/o. Personally though, I'd rather not block hotlinking. But I understand why some people are against it.
  - devicenull 13y ago
    It's actually implemented in a better way (no db required) with most web servers: http://wiki.nginx.org/HttpSecureLinkModule http://wiki.nginx.org/HttpSecureLinkModule
tomjen3 13y ago
No. The web is about links, if you don't want things to be linked too, don't make them accessible to the general public.
- krapp 13y ago
  You can responsibly link to an image on someone else's server without embedding it directly on your site. The web is no longer just hypertext, and the potential inequity in resources and scalability between servers matters. Why should I be expected to suffer and pay for anyone who decides they like a particular image on my site and want to use it in their forum avi? They can just as easily copy it locally and use it.

liotier 13y ago

> The article makes it sound like every webmaster is entitled to see Referer headers. [..] Why do you think it matters whether I came to your site via Google, Reddit, HN or someone else's blog ? As an occasional small-time blogger, I like to know where the conversation happens, where people are interested - it enhances my engagement with my readers. As a reader, I'll gladly grant webmasters that courtesy.

pbreit 13y ago

Uh...I don't think anyone, OP included, is suggesting it's a right. Just that it used to be almost always available. And it's going away, sort of accidentally. On the scale of "useful things" vs "privacy invasions", it's pretty safely on the former side.