6 ms·

Source code: https://gist.github.com/jashkenas/9a4d9e648eb11aab394e https://gist.github.com/jashkenas/9a4d9e648eb11aab394e

dreen 11y ago
Could you give more information on how do some websites react to a google referer?
- patio11 11y ago
  The design intent is almost certainly using Google's "first click free" policy, which primarily applies to newspapers like the NYT or WSJ. Google back in the day struck a deal with the print media: they would remain in Google's index if and only if clicking from a Google search to their articles circumvented their paywall on the first click. If they put only partial content or a full paywall, they'd be delisted. It was not exactly a secret that this would mean that Google would rank scrapers higher for NYT articles than they'd rank the NYT. Thus, when someone posts a paywalled link to HN, someone inevitably posts the advice "Google the title and click through." (Other than, say, "If you're a working professional, just buy a WSJ and NYT subscription.")
  - Mahn 11y ago
    So it's not so much that they intended to have a "first click free" policy in the first place but that they are forced to do an exception in order to remain in Google's index. That's interesting, I always wondered why didn't they apply the same policy to being linked in Reddit or HN.
    - donohoe 11y ago
      Actually, some do. NYT for example allowed (maybe they still do) incoming links from Twitter & FB to bypass paywall
  - colinbartlett 11y ago
    I am a working professional but I have always resisted buying a subscription because I don't want one single publication to become my de facto news source. I would love if I could pay $20 to $30 per month to some aggregator that lets me read articles from Al Jazeera, NY Times, WSJ, BBC, and a selection of other diverse sources.
    - stevesearer 11y ago
      I like Circa News (take my money please!) because they aggregate news from a variety of sources and present news items in small bit sized chunks listing facts or statements. They also link to the appropriate sources. edit: and you can follow news stories to get updates when new information is available.
      - jerrac 11y ago
        Never heard of Circa News before, thanks for mentioning it.
        stevesearer 11y ago
        https://circanews.com/ https://circanews.com/ for the interested. I use the app exclusively though and haven't spent much time on the website.
      - aaronbrethorst 11y ago
        They're either shutting down or being acquired soon: http://fortune.com/2015/04/30/news-app-circa-buyer/ http://fortune.com/2015/04/30/news-app-circa-buyer/
        stevesearer 11y ago
        That's unfortunate. I'd pay for it even if it were free otherwise because I like the service so much.
        malnourish 11y ago
  - wodenokoto 11y ago
    Weren't they the same publishers that were suing google for linking directly to their articles?
    - donohoe 11y ago
      No. They weren't. Some mis-guided ones tried.
  - ascagnel_ 11y ago
    One issue with the subscriptions, at least with the NYT, is that their pricing is a little crazy. The phone version is $15, the tablet version is $20, and both together are $30. It seems... excessive. WSJ at least (used to?) offer a single subscription that gives you access on all devices.
    - acdha 11y ago
      Not to defend the NYT craziness, but that plan somewhat exists in the weekend print edition and should save you money, too. Subscribe, check the box in your account to have them donate your paper copy to a school or library, and enjoy access on all of your devices.
      - Tenhundfeld 11y ago
        I didn't see that option when I signed up a few years ago. When I look at my account now, the only place I see that option is when requesting Vacation Suspension. Do you happen to know if there's a place on the website to set this indefinitely? I guess I can call.
        acdha 11y ago
        I'm on permanent vacation as far as the NYT knows
        Tenhundfeld 11y ago
        Gotcha, and it sounds like that doesn't affect your digital access. Thanks for the tip. At least 50% of time, we just throw away our physical Sunday paper without ever looking at it. It's ridiculous, but I couldn't find any friends who wanted it.
- lfuller 11y ago
  Many news websites will pop up a paywall if you visit a link directly, but won't if you arrive from a Google search.
  - wlesieutre 11y ago
    The reason for this being that Google results penalize sites that show content to the crawler and then hide it from people who visit the page (ie non-subscribers). If they want it in the index, it has to be actually viewable. The alternative would be to not show paywalled content to Google's crawlers, and have them entirely left out of search results.
- deleted 11y ago
  [deleted]
- amyjess 11y ago
  I've seen some websites highlight your search terms for you. It's kinda useful, but it's also kinda creepy at the same time.
kuschku 11y ago
Did someone make a Firefox version yet?
- threedaymonk 11y ago
  You can achieve the same thing with the RefControl add-on, which allows you to set a custom referrer for a particular site: https://addons.mozilla.org/en-US/firefox/addon/refcontrol https://addons.mozilla.org/en-US/firefox/addon/refcontrol
  - yoasif_ 11y ago
    Fantastic tip. For anyone interested in trying this out, use http://news.google.com http://news.google.com as the "custom" referer. Screenshots and help here: http://www.renjusblog.com/2010/01/hack-read-wall-street-online-free.html http://www.renjusblog.com/2010/01/hack-read-wall-street-onli...
    - Quanticles 11y ago
      Thanks, this worked perfectly
    - asquabventured 11y ago
      Perfect! Thank you!
- eCa 11y ago
  For sites that has x free articles/month, I just discovered Private Tab[1] which makes it easy to turn an existing tab private or force a bookmark to always open in a private tab. [1] https://addons.mozilla.org/en-us/firefox/addon/private-tab/ https://addons.mozilla.org/en-us/firefox/addon/private-tab/
  - asquabventured 11y ago
    Self-Destructing cookies extension also keeps the read count at 0 after you close the tab. As an added bonus the extension should also help prevent you from being tracked across the web by various advertising/state agencies.

dreen 11y ago

Could you give more information on how do some websites react to a google referer?

patio11 11y ago
The design intent is almost certainly using Google's "first click free" policy, which primarily applies to newspapers like the NYT or WSJ. Google back in the day struck a deal with the print media: they would remain in Google's index if and only if clicking from a Google search to their articles circumvented their paywall on the first click. If they put only partial content or a full paywall, they'd be delisted. It was not exactly a secret that this would mean that Google would rank scrapers higher for NYT articles than they'd rank the NYT. Thus, when someone posts a paywalled link to HN, someone inevitably posts the advice "Google the title and click through." (Other than, say, "If you're a working professional, just buy a WSJ and NYT subscription.")
- Mahn 11y ago
  So it's not so much that they intended to have a "first click free" policy in the first place but that they are forced to do an exception in order to remain in Google's index. That's interesting, I always wondered why didn't they apply the same policy to being linked in Reddit or HN.
  - donohoe 11y ago
    Actually, some do. NYT for example allowed (maybe they still do) incoming links from Twitter & FB to bypass paywall
- colinbartlett 11y ago
  I am a working professional but I have always resisted buying a subscription because I don't want one single publication to become my de facto news source. I would love if I could pay $20 to $30 per month to some aggregator that lets me read articles from Al Jazeera, NY Times, WSJ, BBC, and a selection of other diverse sources.
  - stevesearer 11y ago
    I like Circa News (take my money please!) because they aggregate news from a variety of sources and present news items in small bit sized chunks listing facts or statements. They also link to the appropriate sources. edit: and you can follow news stories to get updates when new information is available.
    - jerrac 11y ago
      Never heard of Circa News before, thanks for mentioning it.
      - stevesearer 11y ago
        https://circanews.com/ https://circanews.com/ for the interested. I use the app exclusively though and haven't spent much time on the website.
    - aaronbrethorst 11y ago
      They're either shutting down or being acquired soon: http://fortune.com/2015/04/30/news-app-circa-buyer/ http://fortune.com/2015/04/30/news-app-circa-buyer/
      - stevesearer 11y ago
        That's unfortunate. I'd pay for it even if it were free otherwise because I like the service so much.
      - malnourish 11y ago
- wodenokoto 11y ago
  Weren't they the same publishers that were suing google for linking directly to their articles?
  - donohoe 11y ago
    No. They weren't. Some mis-guided ones tried.
- ascagnel_ 11y ago
  One issue with the subscriptions, at least with the NYT, is that their pricing is a little crazy. The phone version is $15, the tablet version is $20, and both together are $30. It seems... excessive. WSJ at least (used to?) offer a single subscription that gives you access on all devices.
  - acdha 11y ago
    Not to defend the NYT craziness, but that plan somewhat exists in the weekend print edition and should save you money, too. Subscribe, check the box in your account to have them donate your paper copy to a school or library, and enjoy access on all of your devices.
    - Tenhundfeld 11y ago
      I didn't see that option when I signed up a few years ago. When I look at my account now, the only place I see that option is when requesting Vacation Suspension. Do you happen to know if there's a place on the website to set this indefinitely? I guess I can call.
      - acdha 11y ago
        I'm on permanent vacation as far as the NYT knows
        Tenhundfeld 11y ago
        Gotcha, and it sounds like that doesn't affect your digital access. Thanks for the tip. At least 50% of time, we just throw away our physical Sunday paper without ever looking at it. It's ridiculous, but I couldn't find any friends who wanted it.
lfuller 11y ago
Many news websites will pop up a paywall if you visit a link directly, but won't if you arrive from a Google search.
- wlesieutre 11y ago
  The reason for this being that Google results penalize sites that show content to the crawler and then hide it from people who visit the page (ie non-subscribers). If they want it in the index, it has to be actually viewable. The alternative would be to not show paywalled content to Google's crawlers, and have them entirely left out of search results.
deleted 11y ago
[deleted]
amyjess 11y ago
I've seen some websites highlight your search terms for you. It's kinda useful, but it's also kinda creepy at the same time.

kuschku 11y ago

Did someone make a Firefox version yet?

threedaymonk 11y ago
You can achieve the same thing with the RefControl add-on, which allows you to set a custom referrer for a particular site: https://addons.mozilla.org/en-US/firefox/addon/refcontrol https://addons.mozilla.org/en-US/firefox/addon/refcontrol
- yoasif_ 11y ago
  Fantastic tip. For anyone interested in trying this out, use http://news.google.com http://news.google.com as the "custom" referer. Screenshots and help here: http://www.renjusblog.com/2010/01/hack-read-wall-street-online-free.html http://www.renjusblog.com/2010/01/hack-read-wall-street-onli...
  - Quanticles 11y ago
    Thanks, this worked perfectly
  - asquabventured 11y ago
    Perfect! Thank you!
eCa 11y ago
For sites that has x free articles/month, I just discovered Private Tab[1] which makes it easy to turn an existing tab private or force a bookmark to always open in a private tab. [1] https://addons.mozilla.org/en-us/firefox/addon/private-tab/ https://addons.mozilla.org/en-us/firefox/addon/private-tab/
- asquabventured 11y ago
  Self-Destructing cookies extension also keeps the read count at 0 after you close the tab. As an added bonus the extension should also help prevent you from being tracked across the web by various advertising/state agencies.