19 ms·
Wait, Google Sent Me
- jashkenas 11y agoSource code: https://gist.github.com/jashkenas/9a4d9e648eb11aab394e https://gist.github.com/jashkenas/9a4d9e648eb11aab394e
- dreen 11y agoCould you give more information on how do some websites react to a google referer?
- patio11 11y agoThe design intent is almost certainly using Google's "first click free" policy, which primarily applies to newspapers like the NYT or WSJ. Google back in the day struck a deal with the print media: they would remain in Google's index if and only if clicking from a Google search to their articles circumvented their paywall on the first click. If they put only partial content or a full paywall, they'd be delisted. It was not exactly a secret that this would mean that Google would rank scrapers higher for NYT articles than they'd rank the NYT. Thus, when someone posts a paywalled link to HN, someone inevitably posts the advice "Google the title and click through." (Other than, say, "If you're a working professional, just buy a WSJ and NYT subscription.")
- Mahn 11y agoSo it's not so much that they intended to have a "first click free" policy in the first place but that they are forced to do an exception in order to remain in Google's index. That's interesting, I always wondered why didn't they apply the same policy to being linked in Reddit or HN.
- donohoe 11y agoActually, some do. NYT for example allowed (maybe they still do) incoming links from Twitter & FB to bypass paywall
- colinbartlett 11y agoI am a working professional but I have always resisted buying a subscription because I don't want one single publication to become my de facto news source. I would love if I could pay $20 to $30 per month to some aggregator that lets me read articles from Al Jazeera, NY Times, WSJ, BBC, and a selection of other diverse sources.
- stevesearer 11y agoI like Circa News (take my money please!) because they aggregate news from a variety of sources and present news items in small bit sized chunks listing facts or statements. They also link to the appropriate sources. edit: and you can follow news stories to get updates when new information is available.
- jerrac 11y agoNever heard of Circa News before, thanks for mentioning it.
- stevesearer 11y agohttps://circanews.com/ https://circanews.com/ for the interested. I use the app exclusively though and haven't spent much time on the website.
- aaronbrethorst 11y agoThey're either shutting down or being acquired soon: http://fortune.com/2015/04/30/news-app-circa-buyer/ http://fortune.com/2015/04/30/news-app-circa-buyer/
- stevesearer 11y agoThat's unfortunate. I'd pay for it even if it were free otherwise because I like the service so much.
- malnourish 11y ago
- wodenokoto 11y agoWeren't they the same publishers that were suing google for linking directly to their articles?
- donohoe 11y agoNo. They weren't. Some mis-guided ones tried.
- ascagnel_ 11y agoOne issue with the subscriptions, at least with the NYT, is that their pricing is a little crazy. The phone version is $15, the tablet version is $20, and both together are $30. It seems... excessive. WSJ at least (used to?) offer a single subscription that gives you access on all devices.
- acdha 11y agoNot to defend the NYT craziness, but that plan somewhat exists in the weekend print edition and should save you money, too. Subscribe, check the box in your account to have them donate your paper copy to a school or library, and enjoy access on all of your devices.
- Tenhundfeld 11y agoI didn't see that option when I signed up a few years ago. When I look at my account now, the only place I see that option is when requesting Vacation Suspension. Do you happen to know if there's a place on the website to set this indefinitely? I guess I can call.
- acdha 11y agoI'm on permanent vacation as far as the NYT knows
- Tenhundfeld 11y agoGotcha, and it sounds like that doesn't affect your digital access. Thanks for the tip. At least 50% of time, we just throw away our physical Sunday paper without ever looking at it. It's ridiculous, but I couldn't find any friends who wanted it.
- lfuller 11y agoMany news websites will pop up a paywall if you visit a link directly, but won't if you arrive from a Google search.
- wlesieutre 11y agoThe reason for this being that Google results penalize sites that show content to the crawler and then hide it from people who visit the page (ie non-subscribers). If they want it in the index, it has to be actually viewable. The alternative would be to not show paywalled content to Google's crawlers, and have them entirely left out of search results.
- deleted 11y ago[deleted]
- amyjess 11y agoI've seen some websites highlight your search terms for you. It's kinda useful, but it's also kinda creepy at the same time.
- kuschku 11y agoDid someone make a Firefox version yet?
- threedaymonk 11y agoYou can achieve the same thing with the RefControl add-on, which allows you to set a custom referrer for a particular site: https://addons.mozilla.org/en-US/firefox/addon/refcontrol https://addons.mozilla.org/en-US/firefox/addon/refcontrol
- yoasif_ 11y agoFantastic tip. For anyone interested in trying this out, use http://news.google.com http://news.google.com as the "custom" referer. Screenshots and help here: http://www.renjusblog.com/2010/01/hack-read-wall-street-online-free.html http://www.renjusblog.com/2010/01/hack-read-wall-street-onli...
- Quanticles 11y agoThanks, this worked perfectly
- asquabventured 11y agoPerfect! Thank you!
- eCa 11y agoFor sites that has x free articles/month, I just discovered Private Tab[1] which makes it easy to turn an existing tab private or force a bookmark to always open in a private tab. [1] https://addons.mozilla.org/en-us/firefox/addon/private-tab/ https://addons.mozilla.org/en-us/firefox/addon/private-tab/
- asquabventured 11y agoSelf-Destructing cookies extension also keeps the read count at 0 after you close the tab. As an added bonus the extension should also help prevent you from being tracked across the web by various advertising/state agencies.
- jebronie 11y agoGoogle users are priviliged for a reason.
- sp332 11y agoI thought Google users were only privileged because news sites want the full text of the page to show up in searches, and Google punishes pages that show different content to the web crawler than they show to searchers.
- cissou 11y agoand the reason is: if a user bounces off a SEO result, the ranking is degraded, and nobody wants that, so Google users can skip the paywalls. I don't think that's a good reason, or even a good way to protect content.
- jfoster 11y agoIt would be nice if Google could make some of the Chrome permissions a bit more fine-grained. This one needs to "read and change your data on all websites", but with a better extensions architecture could be something much less scary like "read and change your data on the current website whenever you click the extension's icon".
- Drakim 11y agoYeah, especially that last part. It would be great if all permissions started with "Upon page load..." or "Upon clicking the extension icon..."
- semenko 11y agoThat granularity exists -- it's called activeTab: https://developer.chrome.com/extensions/activeTab https://developer.chrome.com/extensions/activeTab (Though it might not be possible to modify the referer with activeTab alone :/)
- jashkenas 11y agoNifty tip. Let's try it, and see if it works. Edit: Sorry bud. I gave it a shot, and `activeTab` doesn't allow you to modify the "Referer" header.
- semenko 11y agoYeah, if you really want to do this, you can use chrome.permissions with "optional_permissions" set in your manifest requesting all urls -- then call permissions.request() on invocation to add a specific host. See: https://developer.chrome.com/extensions/permissions https://developer.chrome.com/extensions/permissions
- topynate 11y agoYou could dynamically (and temporarily - use chrome.permissions.remove when the tab navigates or closes) request the webRequest permission for the current host when the user clicks the browser action icon. Yes, that would mean clicking again to confirm whenever the user wants to use the extension, although it's possible that Chrome will remember what permissions have been granted and not prompt the user for future use on the same site.
- joshtaku 11y agoI think it's funny that Jeremy made this considering his background.
- dsjoerg 11y agoyeah i came here to say that
- k-mcgrady 11y agoI was curious so I googled him and saw he created Coffeescript, backbone.js and underscore.js. Is there part of his background I'm missing that makes this funny. Seriously I don't get it and it's bugging me :)
- joshtaku 11y agoThe backbone.js stuff was during NYT work if I recall.
- k-mcgrady 11y agoAh ok, that is pretty funny.
- jashkenas 11y agoThat's not quite correct. Both Backbone.js and Underscore.js were extracted from DocumentCloud — a nonprofit organization. But I am currently a Graphics Editor at the NYT...
- alextgordon 11y agoNot for much longer, I bet...
- jashkenas 11y agoHa ha... We'll find out soon, won't we.
- 11y ago
- drzaiusapelord 11y agoSo change useragent to googlebot?
- toxicFork 11y agoNo: https://news.ycombinator.com/item?id=9531943 https://news.ycombinator.com/item?id=9531943
- udev 11y agoGood idea, but does not work with Financial Times.
- PhantomGremlin 11y agoFT has (as far as I've been able to determine) a hard paywall. But they do give you a certain number of free articles per month (8 IIRC?) if you create an account. Also everything at FT Alphaville is free with an account. Good enough for me!
- pornel 11y agoHi! I work for Financial Times. We've started experimenting with First Click Free — soon you'll get one article a day for free if you come from Twitter, Facebook, LinkedIn or Reddit. We'd like to let Google index more of our content (which will require allowing First Click Free from Google as well), but that depends on our other efforts, such as launch of a redesigned desktop site (you should try the beta, it's pretty cool) and technology changes behind mobile sites. Of course we'll keep First Click Free if it leads to more readers subscribing rather than just faking referrers.
- asquabventured 11y agoCan you add Drudgereport to that list? I follow a lot of links from drudge and when he links to a FT article and I click without realizing I end up at the paywall splash screen and just immediately close the tab.
- higherpurpose 11y agoReally? With a Google™ in the name, too? What do you think is going to happen in the next two hours? 1) They'll take down your extension because you're violating their ToS and their trademark 2) The extension will already piss off their "partners", but they can't really say that's the reason they're taking it down, so good thing you provided them with option 1)! I have nothing against the extension itself. I search for such headlines all the time in Google, too. I just think it's reckless to taunt Google like this and give them an excuse to shut it down on a silver platter.
- k-mcgrady 11y agoIf you use a trademarked name you're supposed to put the tm there. e.g. I used F1 in the name of an app and was contacted by the rights holders. Their only demand was that anywhere I use the text 'F1' I put the tm mark.
- icebraining 11y agoI don't see how he is violating the trademark (though that ™ should be a ®).
- CoryG89 11y agoIt seems like Google recommends using ™ for most of their trademarks. Their website says that laws about using ® vary from country to country and there are sometimes harsh penalties for using it with an unregistered trademark. It seems at least OK to use ™ if you are unsure. http://www.google.com/permissions/trademark/our-trademarks.html http://www.google.com/permissions/trademark/our-trademarks.h... http://www.google.com/permissions/trademark/rules.html http://www.google.com/permissions/trademark/rules.html
- dragonwriter 11y ago> I don't see how he is violating the trademark AFAIK, without a trademark license, using it to promote your product (including using it as part of your product name) is generally a violation of trademark. Nominal fair use allows you to use the trademark to refer to the thing that trademark names in descriptive text, but not generally as part of the name of your product.
- ergest 11y agoThis is gonna mess up some major web analytics stats about referrers :)
- at-fates-hands 11y agoAs someone who works with SEO, this was my first reaction. It's going to make a lot of my analytics wonky and the numbers won't be very accurate. I guess you have to assume such a small percentage of people will actually be using this, so the numbers won't be too bad??
- forgottenpass 11y agoI guess you have to assume such a small percentage of people will actually be using this, so the numbers won't be too bad?? You don't "have to" assume that, but why start worrying about the source of error in your metrics now? The cumulative wight of this and all the other problems with your data probably won't dip their utility below zero, so just go back to plugging your ears and yelling "lalalalala".
- scrollaway 11y agoAs someone who works with SEO, maybe it's time you learn your entire field of work depends on the good will of both search engines and users to not put a screwdriver in the ball you play with.
- at-fates-hands 11y agoActually, I do more development work than SEO, so referring to my "entire field of work" is a bit. . .well. . .erroneous. But hey, whatever makes you feel better pal.
- PeterisP 11y agoIt's the same thing as with browser identification and detection of mobile devices - as soon as you start using them for non-info purposes e.g. disabling content or redirecting people to a mobile-specific page (or an advert for 'app of your site) that they don't want, then you have to expect that any well-meaning software will have the user device lie to you, because it's in the best interests of the user.
- donohoe 11y agoDisappointed in you Jeremy. Its fair for many here to complain about some of the online advertising practices of publishers. However the viable alternative it to pivot to some form of subscription model. Its a hard transition and many orgs are making risky changes to do better. Extensions like these are a step backwards. If you value the work you should support it. See bio for usual disclaimers
- meritt 11y agoCan you share any feedback on the demographics of subscribers who successfully convert via paywalls? I'm curious how well it works with the 18-25 and 26-35 demos. Obviously this is a bit of an echo chamber here but I think a lot of us become annoyed and mentally boycott those sites instead of succumbing to pressure and paying. Have you ever considered doing cross-promotions through other channels, such as the ability to round-up my Starbucks purchases and have the excess go toward a subscription?
- yuncun 11y agoHow do you feel about Adblock?
- patcon 11y agoNot a fan of the shaming... If news orgs value their journalists and the future of the profession, they should experiment with more daring models, rather than ones that simply maximally enrich the org under the same old model EDIT: imho a model that falls apart legally under a browser extension that fits in a 40-line gist is nothing on which to base the livelihoods of a bunch of people
- donohoe 11y agoI agree - but to be fair you're assuming they haven't experimented. Micro-payments, dumb 'capthcha' based models, memberships, events, "premier access". All have been tried in various flavors. Some with limited success. I'm not asking for you to dream up an alternative solution, but don't assume this is a problem that has not been given a lot of thought by many organizations. This is partly why we are stuck with things like "native advertising" and pretty shit IAB ad units.
- javiercr 11y agoIMHO rendering totally different things depending on the http referrer should be considered as cloaking. It's basically the same practice than rendering a different thing if the User-Agent matches Googlebot. I wonder what would happen to your site if you did this without being The New York Times. Kudos to Jeremy for this.
- MichaelGG 11y agoGoogle actually encourages this and call it first click free.
- jrochkind1 11y agoI need a 'facebook sent me' version, lots of websites allow facebook references but not neccesarily google. I'm reduced to going to facebook and sending a message to myself with the url so I can click on it.
- jashkenas 11y ago> lots of websites allow facebook references but not neccesarily google Like what?
- cmstoken 11y agoCurious as well... Which sites?
- jrochkind1 11y agoLooks like most of them ALSO allow a google user-agent, testing with this plugin, so I'm good. Mostly various newspaper sites.
- 0xff00 11y agoPretty sure it's easy to do since this one is open source.
- habosa 11y agoInteresting. I posted a Chrome extension that did the same thing a while back (WSJ specific) and HN unanimously told me I was in the wrong, so I took it down. https://news.ycombinator.com/item?id=8109987 https://news.ycombinator.com/item?id=8109987 This time the comments seem to be mostly positive.
- libria 11y agoApparently having a few javascript frameworks under your belt can put you on the right side of the law.
- manghoti 11y agoNow there's a stinging criticism of HN. Sorry habosa. You're not popular enough to do disruptive things.
- ryandvm 11y agoToo true. I'd love to see what Hacker News comments would look like if all comments were anonymous for the first 6 hours...
- patcon 11y agoHm. That is an unbelievably charming idea for an experiment. Does HN ever do April Fools' day experiments? That would seem worth it for the introspective value alone...
- malnourish 11y agoI think you should suggest this idea to the admins. I know PG stepped down but this seems like something he might have been in favor of.
- bovermyer 11y agoI really like this idea.
- 11y ago
- asquabventured 11y agoJeremy, Useful plugin but BLAH chrome! Any chance you will port this to Firefox?
- umurkontaci 11y agoI have actually made a bookmarklet that allows you to do the same without installing an extension. https://umur.io/paywall-bypass-bookmarklet-come-from-google/ https://umur.io/paywall-bypass-bookmarklet-come-from-google/ Related HN discussion: https://news.ycombinator.com/item?id=9134118 https://news.ycombinator.com/item?id=9134118
- pkaye 11y agoThat is pretty slick. I prefer something lightweight like this.
- jashkenas 11y agoVery nice approach. I didn't know that Google had that auto-redirect feature. But it's not working reliably for me. Frequently, when I click the bookmarklet, it gets stuck on a blank page. Does it work every time for you?
- frik 11y ago> I didn't know that Google had that auto-redirect feature. It uses the functionality of the "I feel lucky" button. It will only work if the site has been indexed by Google, but that's usually the case with news pages that show such fake paywall messages. Btw. please Google fix your frontpage, you serve an unformatted text message (IE9-11): http://s7.postimg.org/52fu9wbdn/google_bug.png http://s7.postimg.org/52fu9wbdn/google_bug.png
- umurkontaci 11y ago
- strommen 11y agoAttention, everybody who publishes on the web: Selectively requiring authentication (based on referrer, user-agent, cookies, etc.) is not a supported feature of the internet. If you want to go ahead and implement it anyway, that's fine - but it's pointless to get mad when it doesn't really work the way you hoped it would.
- wtallis 11y agoMore generally, everyone who publishes on the web should understand that user agent is to be taken literally and at face value. The browser does what the user tells it to do, which may not be what your server asks it to do. A good browser will do things like lie to your server on behalf of the user when you try to get it to act counter to the user's interests.
- xanderjanz 11y agoSure it is. Depends on your definition of authentication. But it's totally a "supported feature of the internet". This plugin isn't fundamentally different than spoofing a session cookie.
- bcg1 11y agoRighteous indignation abounds, a good indicator of achievement. Nice hack jashkenas.
- DigitalSea 11y agoGreat idea. This will probably be up there with the invaluable-ness of Adblock. I have nothing against companies like the Wall Street Journal who want to charge visitors to see content, but if you are going to paywall content at least be consistent and charge everyone or don't charge at all. You can't give it away to visitors from Google and expect people who weren't lucky enough to come from Google to have to pay. It is unfair and discriminatory in my opinion. This kind of reminds me of the aged old trick that people used to use for Google to get better SEO. Normal visitors would get one version of the site, the Googlebot would get an over-SEO optimised version of the site stuffed with keywords and junk (I am talking early 2000's here before Google was as advanced as it is now) to give the impression of a different site. Like I said, I think content publishers should be able to charge people for their content, but you can't have two sets of rules for the same content for the sake of a better SEO ranking and NOT expect people to exploit that loophole. What WSJ and other content publishers are doing is a little borderline dodgy. I am pretty disappointed in the comments from some people here shaming the author for creating this plugin. He isn't doing anything that we haven't already been able to do for a while now. He isn't breaking into servers and getting the content for free, he is exploiting an SEO trick WSJ and other sites are using to avoid being penalised in Google. Completely legitimate if you ask me. If WSJ wanted to truly protect their content behind a paywall, they would have presented the wall to everyone regardless of their referrer.
- Lorento 11y agoYou can. It's a form of price discrimination. Like coupons in a supermarket. People who really want to pay less will jump through the hoop. If you just want it conveniently and don't care about the cost, you can subsidise those others. In a way this is more fair - it means people pay closer to what the money is worth for them.
- cgag 11y agoI'd rather have a plugin that just ensured I never went to these sites at all. Or a list to drop in my hosts file.
- dredmorbius 11y agoWhile I don't have a list of paywall sites, I've been compiling one of Weaponized Viral Clickbait sites generally and the Spartz network (see the New Yorker's profile of the "Virologist" from a few months back) specifically. This is in addition to a set of blocklists curated from the uMatrix Chrome extension. 0.0.0.0 knowd.com # WVC 0.0.0.0 www.greenmedinfo.com # WVC 0.0.0.0 www.heavy.com # WVC 0.0.0.0 freepatriot.org # WVC 0.0.0.0 www.uproxx.com # WVC 0.0.0.0 tcr.tynt.com # WVC 0.0.0.0 slightlyviral.com # WVC 0.0.0.0 pro.moneymappress.com # WVC 0.0.0.0 pro1.moneymappress.com # WVC 0.0.0.0 weknowmemes.com # WVC 0.0.0.0 www.shareable.net # WVC 0.0.0.0 www.thisblewmymind.com # WVC 0.0.0.0 www.tsu.co # WVC 0.0.0.0 www.upworthy.com # WVC 0.0.0.0 www.distractify.com # WVC 0.0.0.0 news.distractify.com # WVC 0.0.0.0 www.evwow.com # WVC 0.0.0.0 evwow.com # WVC 0.0.0.0 www.wimp.com # WVC 0.0.0.0 livelol.me # WVC 0.0.0.0 www.scoopwhoop.com # WVC 0.0.0.0 bustle.com # WVC http://www.bustle.com/articles/62365-there-are-no-words-for-how-horrifying-this-unknown-creature-found-in-a-tuna-can-is 0.0.0.0 www.bustle.com # WVC 0.0.0.0 www.tickld.com # WVC 0.0.0.0 www.zmescience.com # WVC 0.0.0.0 www.addictinginfo.org # WVC 0.0.0.0 www.viralnova.com # WVC 0.0.0.0 opter.co # WVC 0.0.0.0 www.sliptalk.com # WVC https://plus.google.com/u/1/+IrreverentMonk/posts/cuNpUdErc7V 0.0.0.0 sliptalk.com # WVC # Spartz: http://www.newyorker.com/magazine/2015/01/05/virologist 0.0.0.0 www.brainwreck.com # Spartz network 0.0.0.0 www.dose.com # Spartz network 0.0.0.0 www.unfriendable.com # Spartz network 0.0.0.0 www.smartphowned.com # Spartz network 0.0.0.0 smartphowned.com # Spartz network 0.0.0.0 www.givesmehope.com # Spartz network 0.0.0.0 givesmehope.com # Spartz network 0.0.0.0 www.philosoraptors.com # Spartz network 0.0.0.0 philosoraptors.com # Spartz network 0.0.0.0 www.lolbrary.com # Spartz network 0.0.0.0 www.omgfacts.com # Spartz network 0.0.0.0 notsafeforwallet.net # Spartz network 0.0.0.0 sex.omgfacts.com # Spartz network 0.0.0.0 www.itsamememario.com # Spartz network 0.0.0.0 www.whenuseeit.com # Spartz network 0.0.0.0 memes.mugglenet.com # Spartz network 0.0.0.0 www.mugglenet.com # Spartz network 0.0.0.0 www.memeslanding.com # Spartz network 0.0.0.0 www.pokestache.com # Spartz network 0.0.0.0 www.ragestache.com # Spartz network 0.0.0.0 www.sixbillionsecrets.com # Spartz network 0.0.0.0 smartphowned.hollywood.com # Spartz network 0.0.0.0 www.hollywood.com # Spartz network 0.0.0.0 www.tasteofawesome.com # Spartz network 0.0.0.0 www.thatssotrue.com # Spartz network 0.0.0.0 unfriendable.hollywood.com # Spartz network 0.0.0.0 love.givesmehope.com # Spartz network
- digi_owl 11y agoBeen pulling something similar with Firefox and Modify Headers, only i claim i come from Twitter...
- dredmorbius 11y agoThis seems like a feature which Privoxy could provide... http://www.privoxy.org/ http://www.privoxy.org/
- manymany 11y agoI think they took it down :( guess I'm gonna have to find out how to set it up from source code: https://gist.github.com/jashkenas/9a4d9e648eb11aab394e https://gist.github.com/jashkenas/9a4d9e648eb11aab394e
- jrochkind1 11y agoThis extension seems to have been removed from the Chrome Web Store. Hmmmm.