7 ms·

Oh, I'll really miss occasionally peeking at AWStats and discovering weird pages pointing at my weird pages :( This subtle aspect of web had been always strang

by myfonj 6y ago

Oh, I'll really miss occasionally peeking at AWStats and discovering weird pages pointing at my weird pages :( This subtle aspect of web had been always strangely appealing for me: people leaving trails in access logs and building real "footpaths" network of synapses between HTML documents, across origins. Sad to watch it dying, however beneficial and understandable it is. I feel it didn't have to be this way: maybe if GET wasn't so widely misused recently and generally everybody knew what to not put in URL and acted accordingly, we could have preserved such nice things.

leephillips 6y ago
Totally agree. As almost no one uses anything like webmentions, this was the way to discover who was linking to your site. (And because the Google link tool didn’t seem to return much actual information.)
- nerdponx 6y ago
  Bring it back! https://news.ycombinator.com/item?id=26401955 https://news.ycombinator.com/item?id=26401955
- myfonj 6y ago
  From a glance at MDN [1] and specs [3] it seems at least we can still opt-in our sites to suggest visitors user agent to pass full referrer address along outgoing links; there seems to be three ways to do so: # 1. Apache conf Header set Referrer-Policy "no-referrer-when-downgrade"  <meta name="referrer" content="no-referrer-when-downgrade">  <a href="https://…" referrerpolicy="no-referrer-when-downgrade"> This will pass full path and query when navigating between HTTPS to foreign origin. I guess it will be lost in http: to https: redirects. `unsafe-url` value would do it even for HTTP. (Funny all three have different spelling, and that in effect they direct value of a single HTTP header with inherently erroneous spelling.) [1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referrer-Policy https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Re... [2] my guess using `<meta http-equiv="Referrer-Policy" content="no-referrer-when-downgrade">` should do the same, but is not explicitly mentioned anywhere. [3] https://w3c.github.io/webappsec-referrer-policy/ https://w3c.github.io/webappsec-referrer-policy/
  - simias 6y ago
    Note that it only works if it's set on the source page, for obvious reasons. So if you set these headers on your page the browser will send the referrer when browsing away from your page, but not when your page is the target.
ehsankia 6y ago
Absolutely, I loved seeing spikes in my traffic and going to the reddit/HN thread that caused it. I guess you still can manually figure it out using a search engine maybe, but like you said the smaller weird ones will go under the cover probably.
- kevincox 6y ago
  Exactly. I am torn on this feature. On one hand this is the correct default for privacy, there are a number of websites with somewhat sensitive information in the URL. But relying on a search engine or webmaster tools to crawl the entire web to see where your traffic is coming from is a real shame. Especially because it will have an inevitable delay.
  - gscott 6y ago
    The next step is not knowing where your visitors are coming from and having to only buy google ads rather then creating affiliate relationships. It is too bad FireFox is playing into this, it is not helping anyone's privacy. It only lines Google's pockets.
    - Dylan16807 6y ago
      Using and needing the exact url to form an affiliate relationship sounds like a very rare situation. And I don't know how you can say it doesn't help privacy with a straight face.
      - gscott 6y ago
        It makes total sense that you have a website but have no idea where people are coming from to get to it. Why know what is working for your website so that you can improve upon it?
        Dylan16807 6y ago
        No idea?? You know the domain, which is usually enough by itself. And 99% of the time you can quickly search the domain to find what links to you.
stevenicr 6y ago
Similar here, and as I have written recently: https://news.ycombinator.com/item?id=26532433 https://news.ycombinator.com/item?id=26532433 I feel that this could easily be a user toggle-able option near the url bar - and that people could choose to send referrer and search parameters to sites they like - and I could understand certain medical terms and what-not maybe people wanting to keep more hidden. I'm all for the browser taking control of this and giving the users options - love to be able to have the web site detect this and add a box saying 'hey would you mind sharing referral and or search term with us" It really helps when trying to figure out if you should be adding more content for 'term-mainly-X-demo' might be searching or for sure should be adding more content for other terms people are definitely searching for. I would imagine that many people would like to hide referrer for fbook, most of my sites I would think people would be willing to help with providing such info. as far as webmastering and stats to try to help visitors - it appears google is keeping more and more data to themselves, and now firefox is making it less transparent too. I see there is a link to the referrer policy for your site, but nothing about begging the user to include the info when visiting - oh well. awstats and similar have helped troubleshoot and helped expand many sites for many years and now another nudge to install third party slow-ware google-stats.. I just, sigh.
- behnamoh 6y ago
  > I feel that this could easily be a user toggle-able option near the url bar... No ordinary user wants to deal with that much complexity.
  - skinkestek 6y ago
    But why have we decided lately to fall for the tyranny of what "ordinary users" want? Why can't we have a full featured Firefox hidden behind a setting, so this mythical "ordinary user" can have a dumbed down interface and the rest of us can have a real browser?
    - antonvs 6y ago
      > But why have we decided lately to fall for the tyranny of what "ordinary users" want? For the same reason bank robbers rob banks. That's where the money is. The source of the funding that pays for development of these tools is driving this. I'm not defending that situation, that's just the way it is. "Hackers" working on what they find technically elegant and useful are not driving these sorts of changes.
TheRealPomax 6y ago
Why? What happened here is a change to the default value when https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referrer-Policy https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Re... value if it's left unspecified. Set it to what you want, and this change doesn't affect you?
- zerocrates 6y ago
  It's the policies of the external sites that matter to the parent poster, not their own.
forgotmypw17 6y ago
Yeah, the big sites will still get to see the data though, so if you just become an employee, you can probably still get it.
TriNetra 6y ago
yes, for content-only sites it makes perfect sense. For sites hosting private data requiring login, (especially if there's some sensitive information in the URL) it poses a risk.
simias 6y ago
You'll still get the source domain name, you just won't be able to see the full path. Of course how useful it is will depend on the source website, if it's "news.ycombinator.com" it won't be too difficult to find the post, if it's "facebook.com" it'll be a tad more complicated.
esaym 6y ago
Yes, I came to lament AWStats as well. I remember when google started stripping search terms in their referring urls. So then you didn't know what search terms were pointing to your site. Google's answer was to just get your site verified and use their search console tools. Annoying, but OK. But then other search engines followed suit which left awstats with a lot of missing data. Now with simplified referring urls... it will mostly be a fancy page counter :/
cirno 6y ago
I suspect this is going to be a huge boon for services like ahrefs that crawls the web and provides information on which URLs on the internet link to which other URLs.

by myfonj 6y ago

leephillips 6y ago

Totally agree. As almost no one uses anything like webmentions, this was the way to discover who was linking to your site. (And because the Google link tool didn’t seem to return much actual information.)

nerdponx 6y ago
Bring it back! https://news.ycombinator.com/item?id=26401955 https://news.ycombinator.com/item?id=26401955
myfonj 6y ago
From a glance at MDN [1] and specs [3] it seems at least we can still opt-in our sites to suggest visitors user agent to pass full referrer address along outgoing links; there seems to be three ways to do so: # 1. Apache conf Header set Referrer-Policy "no-referrer-when-downgrade"  <meta name="referrer" content="no-referrer-when-downgrade">  <a href="https://…" referrerpolicy="no-referrer-when-downgrade"> This will pass full path and query when navigating between HTTPS to foreign origin. I guess it will be lost in http: to https: redirects. `unsafe-url` value would do it even for HTTP. (Funny all three have different spelling, and that in effect they direct value of a single HTTP header with inherently erroneous spelling.) [1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referrer-Policy https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Re... [2] my guess using `<meta http-equiv="Referrer-Policy" content="no-referrer-when-downgrade">` should do the same, but is not explicitly mentioned anywhere. [3] https://w3c.github.io/webappsec-referrer-policy/ https://w3c.github.io/webappsec-referrer-policy/
- simias 6y ago
  Note that it only works if it's set on the source page, for obvious reasons. So if you set these headers on your page the browser will send the referrer when browsing away from your page, but not when your page is the target.

ehsankia 6y ago

Absolutely, I loved seeing spikes in my traffic and going to the reddit/HN thread that caused it. I guess you still can manually figure it out using a search engine maybe, but like you said the smaller weird ones will go under the cover probably.

kevincox 6y ago
Exactly. I am torn on this feature. On one hand this is the correct default for privacy, there are a number of websites with somewhat sensitive information in the URL. But relying on a search engine or webmaster tools to crawl the entire web to see where your traffic is coming from is a real shame. Especially because it will have an inevitable delay.
- gscott 6y ago
  The next step is not knowing where your visitors are coming from and having to only buy google ads rather then creating affiliate relationships. It is too bad FireFox is playing into this, it is not helping anyone's privacy. It only lines Google's pockets.
  - Dylan16807 6y ago
    Using and needing the exact url to form an affiliate relationship sounds like a very rare situation. And I don't know how you can say it doesn't help privacy with a straight face.
    - gscott 6y ago
      It makes total sense that you have a website but have no idea where people are coming from to get to it. Why know what is working for your website so that you can improve upon it?
      - Dylan16807 6y ago
        No idea?? You know the domain, which is usually enough by itself. And 99% of the time you can quickly search the domain to find what links to you.

stevenicr 6y ago

Similar here, and as I have written recently: https://news.ycombinator.com/item?id=26532433 https://news.ycombinator.com/item?id=26532433 I feel that this could easily be a user toggle-able option near the url bar - and that people could choose to send referrer and search parameters to sites they like - and I could understand certain medical terms and what-not maybe people wanting to keep more hidden. I'm all for the browser taking control of this and giving the users options - love to be able to have the web site detect this and add a box saying 'hey would you mind sharing referral and or search term with us" It really helps when trying to figure out if you should be adding more content for 'term-mainly-X-demo' might be searching or for sure should be adding more content for other terms people are definitely searching for. I would imagine that many people would like to hide referrer for fbook, most of my sites I would think people would be willing to help with providing such info. as far as webmastering and stats to try to help visitors - it appears google is keeping more and more data to themselves, and now firefox is making it less transparent too. I see there is a link to the referrer policy for your site, but nothing about begging the user to include the info when visiting - oh well. awstats and similar have helped troubleshoot and helped expand many sites for many years and now another nudge to install third party slow-ware google-stats.. I just, sigh.

behnamoh 6y ago
> I feel that this could easily be a user toggle-able option near the url bar... No ordinary user wants to deal with that much complexity.
- skinkestek 6y ago
  But why have we decided lately to fall for the tyranny of what "ordinary users" want? Why can't we have a full featured Firefox hidden behind a setting, so this mythical "ordinary user" can have a dumbed down interface and the rest of us can have a real browser?
  - antonvs 6y ago
    > But why have we decided lately to fall for the tyranny of what "ordinary users" want? For the same reason bank robbers rob banks. That's where the money is. The source of the funding that pays for development of these tools is driving this. I'm not defending that situation, that's just the way it is. "Hackers" working on what they find technically elegant and useful are not driving these sorts of changes.

TheRealPomax 6y ago

Why? What happened here is a change to the default value when https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referrer-Policy https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Re... value if it's left unspecified. Set it to what you want, and this change doesn't affect you?

zerocrates 6y ago
It's the policies of the external sites that matter to the parent poster, not their own.

forgotmypw17 6y ago

Yeah, the big sites will still get to see the data though, so if you just become an employee, you can probably still get it.

TriNetra 6y ago

yes, for content-only sites it makes perfect sense. For sites hosting private data requiring login, (especially if there's some sensitive information in the URL) it poses a risk.

simias 6y ago

You'll still get the source domain name, you just won't be able to see the full path. Of course how useful it is will depend on the source website, if it's "news.ycombinator.com" it won't be too difficult to find the post, if it's "facebook.com" it'll be a tad more complicated.

esaym 6y ago

Yes, I came to lament AWStats as well. I remember when google started stripping search terms in their referring urls. So then you didn't know what search terms were pointing to your site. Google's answer was to just get your site verified and use their search console tools. Annoying, but OK. But then other search engines followed suit which left awstats with a lot of missing data. Now with simplified referring urls... it will mostly be a fancy page counter :/

cirno 6y ago

I suspect this is going to be a huge boon for services like ahrefs that crawls the web and provides information on which URLs on the internet link to which other URLs.