25 ms·

Getting around website paywalls with devtools alone

169 points by bebrws 4y ago

bubblebaker 4y ago
My method is to use ublock origin extention to block third party scripts.
bebrws 4y ago
A code free way to view a site with a paywall and navigate or scroll your way through the content.
arbol 4y ago
Good tip. Sites turning off scroll is one of my pet annoyances.
CodeHz 4y ago
Maybe it just a transparent div that blocked all the pointer events?
drewtato 4y ago
Usually on these sites, there'll be an `overflow: hidden` element that's holding all the content. If you can find and disable that CSS line, it'll work as normal. Or just save it to the Wayback Machine and read it through that.
- prettyStandard 4y ago
  Not sure if this is exactly right, but something like this should obviate the need to find the element holding all the content. * { overflow: visible !important; }
- bebrws 4y ago
  I am going to add this to the post if that is alright. Please let me know if I shouldn't cite your HN username on the post. You can reply here and I'll see it. Thank you, this is super helpful.
  - drewtato 4y ago
    Kinda late, but that's okay with me!
_boffin_ 4y ago
or... just remove the `overflow: hidden` that's most likely placed on the `<body>` or some high level `<div>`.
- lol768 4y ago
  This is the correct answer - and then scrolling will work properly! Not sure why this 'hack' is on the front-page.
  - spiritplumber 4y ago
    I learned a thing today that I would not have if it wasn't :)
- courgette 4y ago
  my experience is that it stopped working a few years ago. Eg: NYT or FT. The content is nowhere to be fund on the raw html itself. No idea how it works but it looks like actual content is loaded separately once the gates are open?
start123 4y ago
I just use Firefox's reader view. Does the same with just a click. If it doesn't work, just refresh in reader view and it should load properly.
- bebrws 4y ago
  Added this comment to the post if that is ok. Let me know here if it isn't. Thank you!
batperson 4y ago
In the case of Washington Post it just has a "position: fixed" style on the <body> element. That's usually the case with most of these scroll locking sites, one of the root parent elements will have some CSS style that you can click off.
encryptluks2 4y ago
Some sites now are literally not loading the actual paywalled content until after you sign in, so not matter what you do you aren't going to be able to access it unless someone with a paid subscription shares that content and it is then uploaded to a third party paywall bypasser.
- colesantiago 4y ago
  The Information does this a lot. https://www.theinformation.com https://www.theinformation.com
kozinc 4y ago
Sometimes, however, you don't even get a full article text when the paywalled site loads. In those cases, no amount of Scroll Into View will help. For those cases try something like Google Cache or Wayback Machine! It still won't always work, but it's nevertheless got a pretty good success rate.
samwillis 4y ago
Many sites don't contains the full content even if you do that. I'm not sure of how it works (does it subscribed to them all?) but https://archive.ph/ https://archive.ph/ is a good way to see the content in those cases. But really, if you are regularly reading content on a site you should subscribe to support the journalists employed there.
- vie00001 4y ago
  > I'm not sure of how it works (does it subscribed to them all?) but https://archive.ph/ https://archive.ph/ is a good way to see the content in those cases. I think for search engine crawlers there are versions without a paywall so these articles can get fully indexed. Archive.ph, and similar services, might get the full content this way somehow. But I am just guessing.
  - crakenzak 4y ago
    you're spot on, this is exactly how sites like archive.org, archive.ph, or even if you click on "view cached version" on Google get the non-paywalled versions.
  - maxboone 4y ago
    archive.ph also uses (donated) logins to archive (paywalled) content, however those accounts do get blocked from time to time. https://blog.archive.today/post/678202832257794048/why-cant-instagram-pages-be-archived-anymore https://blog.archive.today/post/678202832257794048/why-cant-... While pretending to be GoogleBot used to get you full articles (or grabbing them from cache) this doesn't seem to be the case for some sites anymore. They just give the first part of the article without the paywall, as that's usually enough for SEO purposes.
    - jhncls 4y ago
      > They just give the first part of the article without the paywall, as that's usually enough for SEO purposes. Many consumers often wouldn't read more text anyway. About one paragraph might even be too much to fill the modern attention span.
kjeksfjes 4y ago
Hush?
karrotwaltz 4y ago
I use this JS bookmarklet to remove fixed elements and restore scrolling, it works most of the time: https://pastebin.com/qBjJHkMv https://pastebin.com/qBjJHkMv I also have one to kill all running javascript and remove all event listeners, it works wonders when you are redirected to a paywall / login page after a few seconds.
- albert_e 4y ago
  Would you mind sharing the second script as well? Thanks This is supposed to be saved as a Javascript Bookmarklet?
  - karrotwaltz 4y ago
    Javascript killer: https://pastebin.com/utE3275J https://pastebin.com/utE3275J Yes, I'm using it as a bookmarklet. I'm using firefox but I think it should work the same for other browsers.
_the_inflator 4y ago
A Clickbait Transformator would have opted for a headline along the lines: "This tip saves you thousands of Dollars!". ;)
PufPufPuf 4y ago
I recommend the browser extension "Bypass Paywalls Clean". I sometimes think about the morality of using it, but I just don't find it viable to pay all the websites where I read just a single article.
- denton-scratch 4y ago
  > but I just don't find it viable to pay all the websites where I read just a single article. This. In the print days, you'd buy a newspaper; you'd have access to all the articles in that edition. I used to read a daily paper. In the modern world, these papers expect you to pay for a newspaper just to read a single article. I dunno, perhaps they could form a "Paywall Consortium", so that I could pay a one-day fee to the consortium, and have access to Washpo, Telegraph, NYT etc. for 24 hours. Let the consortium figure out how to distribute the fees - it's not my concern. But if you want me to buy the whole paper to read a single article, well, ain't gonna happen.
  - Kerrick 4y ago
    > But if you want me to buy the whole paper to read a single article, well, ain't gonna happen. This was common for non-subscribers in the print days. Newspapers would print a number of enticing headlines and images on the front page above the fold, and display those folded newspapers for sale at dispensers, newsstands, and stores. Many people who bought a one-off paper would buy for a single article that interested them.
    - mrkstu 4y ago
      For a quarter. Which I would pay now for a single article that interested me, no physical paper required.
      - denton-scratch 4y ago
        A quarter? For a printed newspaper? The cover price of The Guardian is £2.50 weekdays, £3.50 weekends (about 5 quarters and 8 quarters respectively). Apparently Washpo is $2.50 for the daily edition, which is about two quid.
retox 4y ago
A handful of sites will present the subscribers view of the page if you put a dot after the tld part of the url, i.e. https://site.com./1235/article https://site.com./1235/article Those behind Cloudflare don't seem to be vulnerable to this though. I've emailed the sites I've found where this works and none of them have fixed it after a year.
- neoromantique 4y ago
  >I've emailed the sites I've found where this works Why?
  - bilekas 4y ago
    Can't speak for OP but I think I would be annoyed if I was paying a subscription and I found out I could get around it with a simple `.` because of developer incompetence.
    - neoromantique 4y ago
      If you are paying for a subscription then hopefully you do it to support the publication and find their work useful, and not because of a nag banner.
      - bilekas 4y ago
        > to support the publication and find their work useful, and not because of a nag banner. Unfortunately they kind of go hand in hand these days!
      - z3t4 4y ago
        People don't pay for free stuff, even if they like it. People are however lazy, and many would pay to not have to put a dot in the URL every time.
    - manojlds 4y ago
      Good that you spoke for yourself.
suhaybh 4y ago
I just use https://github.com/iamadamdev/bypass-paywalls-chrome https://github.com/iamadamdev/bypass-paywalls-chrome and it works well for me.
alkonaut 4y ago
10 years ago paywalled sites contained the content just hidden. Today I haven't seen a site in a long time that renders the content hidden (why would it do that? There is no reason to do it based on indexing/SEO as far as I'm aware). Even cached/archived versions these days tend to not include the whole text. Basically: they figured out how to make a paywall, which frankly isn't that surprising.
- donohoe 4y ago
  Not sure I agree with that assessment. There are so many ways to do a paywall and you’ll still see all sorts of flavors across the web today.
jwr 4y ago
What I find annoying about paywalled sites is that they provide the full content to Google. And Google is OK with indexing the full content, even though it is not available on the internet, and even though they explicitly forbid the practice of showing different content to a search engine from what is available publicly. Paywalled sites are just fine, but they are not part of the open Internet, and should not pretend to be.
- CM30 4y ago
  Yeah, 100% agree with this. It's like these sites want to have their cake and eat it, and both get the traffic the 'open web' provides while not having to actually share any of their work there. It's like if you needed an app to view a page, yet Google had all its content indexed. Why is that (rightly) seen as unreasonable while charging users for content you provide to bots for free isn't?
- donohoe 4y ago
  > they explicitly forbid the > practice of showing different > content to a search engine from > what is available publicly This isn’t true. This paywall treatment is something they do allow and have worked to accommodate.
  - tyingq 4y ago
    It's also true that they say "cloaking" isn't allowed, "any type of cloaking" https://youtu.be/QHtnfOgp65Q https://youtu.be/QHtnfOgp65Q
- iicc 4y ago
  Sometimes the full content is available - but only if you navigate to it via a Google results page, and you don't have existing cookies implementing a "free article limit".
- mrobins 4y ago
  That’s a Google problem not a publisher problem. Content should be findable and available to people who want to pay for it. Google could easily add an option to filter out paywalled content but that would reduce clicks.
snaix 4y ago
Shortcut in macos, in most browsers, to get into "div selection mode" is "shift + command + c"... Just press that, select the paywall (and any other junk backdrops/opaque divs), and press delete. Sometimes the site also sets an `overflow: hidden` in the css, and you need to remove that to see the content..
vixen99 4y ago
Did not work with https://www.spectator.co.uk/ https://www.spectator.co.uk/.
vixen99 4y ago
Does not work with https://www.spectator.co.uk/ https://www.spectator.co.uk/
phtrivier 4y ago
Title should be : "getting around very poorly implemented paywall (eg WaPo) with devtools alone". As soon as your site sends the whole content of the article to the browser, you're not even trying seriously. (And Firefox "reading mode" is just much better ux than the devtools.)
1vuio0pswjnm7 4y ago
Another option no one has menitoned is AMP. Many sites that try to use "paywalls" have AMP URLs which point to pages that have all the full text of the article in <p> tags. These AMP sites generally look great in a text-only browser that does not run Javascript. Popular example is WSJ. In the URL, add /amp before /article/. Paywalls are insidious because they target non-subscribers. Why let non-subscribers view articles. Why not password protect all subscriber content. Paywalls are a way to make money from (the attention of) non-subscribers, targeting them with ads and tracking. The strategy is apparently to annoy people to the point of subscribing. Yet even if they subscribe they will still be subjected to advertising. One potential advantage is that a paying subscriber has an enforceable contract. In theory the contract could contain enforceable privacy protections. "Tech" companies would never agree to give people enforceable privacy protections; it would destroy their "business". The way to save journalism, especially local news, is to regulate "Big Tech" middlemen, who generally do not employ journalists and produce zero content.1 The quality of journalism in general has taken a nosedive, but placing the blame for that on web users not purchasing subscriptions is conveniently ignoring the true culprit. 1. Arguably that's a prerequisite to maintaining their Section 230 protection. In the recent Supreme Court oral arguments, Google's counsel argued Google is not a publisher. Then minutes later she argued Google has to make design decisions "like any publisher", therefore Google gets a free pass to reorganise information in annoying and perhaps harmful ways to maximise ad services revenue, like inserting "popular" videos into YouTube search results that have nothing to do with the query string.
- donohoe 4y ago
  But the number of AMP versions is dwindling as Google is no longer forcing it.
  - 1vuio0pswjnm7 4y ago
    Yep. And the browser vendor can make changes to DevTools, archive.ph is already privacy-unfriendly and blocked in some countries, it could disappear, 12ft.io could disappear, and so on. The fact that seems least likely to change is that most "paywalls" rely on Javascript, CSS or some other "feature" of so-called "modern" web browsers. I have lost count of how many times an HN comment alleges "paywalled" in a thread and I am reading the article just like any other because I am not using a graphical web browser. I would not even know there was a "paywall". Paywalls have dependencies.
porbelm 4y ago
Yeah on the system most of the local papers use over here, the full content is not even loaded for non-subscribers. It used to be, so removing the blur div worked, but now only the headline, byline and lead text are visible :( Guess they caught on to the "cheaters."
dspillett 4y ago
> But I didn't know how to scroll down on the page until today. That is usually due to an "overflow: hidden" somewhere near the top of the DOM tree. Remove that and your normal scrollbar usually returns. You see this a lot with "accept being stalked to read this" pop-overs as well as paywall related shenanigans. I've seen some sites put it back in via JS. There is probably a workaround for that too though the easiest one is to not worry about it and DNS blacklist the site so you don't waste time visiting it again in future.
Eupraxias 4y ago
Anyone else just read content in source?
jrberendt 4y ago
It's insane that news sites with thousands or even millions of monthly users have paywalls like this. Sure it's great for anyone who wants to read for free, but some individuals make their livings now through paywalled articles. If the big guys can't protect from exploits, what about everyone else? I created https://turbolink.io https://turbolink.io to attempt to solve this problem.
thallosaurus 4y ago
Oh yeah, used this method to bypass a regional paywalled news site until they fixed it by sending out scrambled text that seems random enough to not be a cipher ("Zc xjixc Axiäclxiqcil jcqlxi ljx Zxcxqjxiqxi cc")
iKlsR 4y ago
The simplest method is just to refresh the page and hit Esc as quickly as you can to cancel the page load and prevent further scripts from running, that works for me 80% of the time. Sites need to be indexed by google so any paywall is often client side js.
kokanee 4y ago
If you're unable to scroll the page, it generally means there is a "position: fixed" css rule on the body or a wrapper element. Turn off that css rule and you can scroll through the article normally. Publishers are slowly wising up, though. Most don't load the full article for unpaid viewers anymore.
pentagrama 4y ago
This extension removes paywalls on many news sites (maintained, 33k stars) https://github.com/iamadamdev/bypass-paywalls-chrome https://github.com/iamadamdev/bypass-paywalls-chrome
- latchkey 4y ago
  There was some previous drama and some people switched to this one. https://gitlab.com/magnolia1234/bypass-paywalls-chrome-clean/ https://gitlab.com/magnolia1234/bypass-paywalls-chrome-clean... https://gitlab.com/magnolia1234/bypass-paywalls-firefox-clean/-/issues/1 https://gitlab.com/magnolia1234/bypass-paywalls-firefox-clea...
oh_sigh 4y ago
Generally just disabling JavaScript for the site works perfectly.
l3mure 4y ago
Another devtools trick I've used is network throttling to allow me to copy out an article's content before JS loads.
71a54xd 4y ago
Your site's feature of "cool comments from HN" is awesome! Automating this with some kind of LLM and selling it would be killer.
Am4TIfIsER0ppos 4y ago
Just like with pointless loading spinners. You just delete the element which is overlaid on the complete content. Sometimes there is an overflow, opacity, or visibility attribute that needs changing. Fucking webdevs!
kleinmatic 4y ago
Pretty amazing how many people's ethics are less valuable to them than their product design opinions.
throwaway29303 4y ago
It seems the cat's out of the bag now. There's also another interesting way of getting around paywalls: a simple race condition. While loading the page hit stop as soon as you load enough of the content you're interested in et voilà. It does not work 100% of the time and I'm not using Chrome, however. ;)

169 points by bebrws 4y ago

bubblebaker 4y ago

My method is to use ublock origin extention to block third party scripts.

bebrws 4y ago

A code free way to view a site with a paywall and navigate or scroll your way through the content.

arbol 4y ago

Good tip. Sites turning off scroll is one of my pet annoyances.

CodeHz 4y ago

Maybe it just a transparent div that blocked all the pointer events?

drewtato 4y ago

Usually on these sites, there'll be an `overflow: hidden` element that's holding all the content. If you can find and disable that CSS line, it'll work as normal. Or just save it to the Wayback Machine and read it through that.

prettyStandard 4y ago
Not sure if this is exactly right, but something like this should obviate the need to find the element holding all the content. * { overflow: visible !important; }
bebrws 4y ago
I am going to add this to the post if that is alright. Please let me know if I shouldn't cite your HN username on the post. You can reply here and I'll see it. Thank you, this is super helpful.
- drewtato 4y ago
  Kinda late, but that's okay with me!

_boffin_ 4y ago

or... just remove the `overflow: hidden` that's most likely placed on the `<body>` or some high level `<div>`.

lol768 4y ago
This is the correct answer - and then scrolling will work properly! Not sure why this 'hack' is on the front-page.
- spiritplumber 4y ago
  I learned a thing today that I would not have if it wasn't :)
courgette 4y ago
my experience is that it stopped working a few years ago. Eg: NYT or FT. The content is nowhere to be fund on the raw html itself. No idea how it works but it looks like actual content is loaded separately once the gates are open?

start123 4y ago

I just use Firefox's reader view. Does the same with just a click. If it doesn't work, just refresh in reader view and it should load properly.

bebrws 4y ago
Added this comment to the post if that is ok. Let me know here if it isn't. Thank you!

batperson 4y ago

In the case of Washington Post it just has a "position: fixed" style on the <body> element. That's usually the case with most of these scroll locking sites, one of the root parent elements will have some CSS style that you can click off.

encryptluks2 4y ago

Some sites now are literally not loading the actual paywalled content until after you sign in, so not matter what you do you aren't going to be able to access it unless someone with a paid subscription shares that content and it is then uploaded to a third party paywall bypasser.

colesantiago 4y ago
The Information does this a lot. https://www.theinformation.com https://www.theinformation.com

kozinc 4y ago

Sometimes, however, you don't even get a full article text when the paywalled site loads. In those cases, no amount of Scroll Into View will help. For those cases try something like Google Cache or Wayback Machine! It still won't always work, but it's nevertheless got a pretty good success rate.

samwillis 4y ago

Many sites don't contains the full content even if you do that. I'm not sure of how it works (does it subscribed to them all?) but https://archive.ph/ https://archive.ph/ is a good way to see the content in those cases. But really, if you are regularly reading content on a site you should subscribe to support the journalists employed there.

vie00001 4y ago
> I'm not sure of how it works (does it subscribed to them all?) but https://archive.ph/ https://archive.ph/ is a good way to see the content in those cases. I think for search engine crawlers there are versions without a paywall so these articles can get fully indexed. Archive.ph, and similar services, might get the full content this way somehow. But I am just guessing.
- crakenzak 4y ago
  you're spot on, this is exactly how sites like archive.org, archive.ph, or even if you click on "view cached version" on Google get the non-paywalled versions.
- maxboone 4y ago
  archive.ph also uses (donated) logins to archive (paywalled) content, however those accounts do get blocked from time to time. https://blog.archive.today/post/678202832257794048/why-cant-instagram-pages-be-archived-anymore https://blog.archive.today/post/678202832257794048/why-cant-... While pretending to be GoogleBot used to get you full articles (or grabbing them from cache) this doesn't seem to be the case for some sites anymore. They just give the first part of the article without the paywall, as that's usually enough for SEO purposes.
  - jhncls 4y ago
    > They just give the first part of the article without the paywall, as that's usually enough for SEO purposes. Many consumers often wouldn't read more text anyway. About one paragraph might even be too much to fill the modern attention span.

kjeksfjes 4y ago

Hush?

karrotwaltz 4y ago

I use this JS bookmarklet to remove fixed elements and restore scrolling, it works most of the time: https://pastebin.com/qBjJHkMv https://pastebin.com/qBjJHkMv I also have one to kill all running javascript and remove all event listeners, it works wonders when you are redirected to a paywall / login page after a few seconds.

albert_e 4y ago
Would you mind sharing the second script as well? Thanks This is supposed to be saved as a Javascript Bookmarklet?
- karrotwaltz 4y ago
  Javascript killer: https://pastebin.com/utE3275J https://pastebin.com/utE3275J Yes, I'm using it as a bookmarklet. I'm using firefox but I think it should work the same for other browsers.

_the_inflator 4y ago

A Clickbait Transformator would have opted for a headline along the lines: "This tip saves you thousands of Dollars!". ;)

PufPufPuf 4y ago

I recommend the browser extension "Bypass Paywalls Clean". I sometimes think about the morality of using it, but I just don't find it viable to pay all the websites where I read just a single article.

denton-scratch 4y ago
> but I just don't find it viable to pay all the websites where I read just a single article. This. In the print days, you'd buy a newspaper; you'd have access to all the articles in that edition. I used to read a daily paper. In the modern world, these papers expect you to pay for a newspaper just to read a single article. I dunno, perhaps they could form a "Paywall Consortium", so that I could pay a one-day fee to the consortium, and have access to Washpo, Telegraph, NYT etc. for 24 hours. Let the consortium figure out how to distribute the fees - it's not my concern. But if you want me to buy the whole paper to read a single article, well, ain't gonna happen.
- Kerrick 4y ago
  > But if you want me to buy the whole paper to read a single article, well, ain't gonna happen. This was common for non-subscribers in the print days. Newspapers would print a number of enticing headlines and images on the front page above the fold, and display those folded newspapers for sale at dispensers, newsstands, and stores. Many people who bought a one-off paper would buy for a single article that interested them.
  - mrkstu 4y ago
    For a quarter. Which I would pay now for a single article that interested me, no physical paper required.
    - denton-scratch 4y ago
      A quarter? For a printed newspaper? The cover price of The Guardian is £2.50 weekdays, £3.50 weekends (about 5 quarters and 8 quarters respectively). Apparently Washpo is $2.50 for the daily edition, which is about two quid.

retox 4y ago

A handful of sites will present the subscribers view of the page if you put a dot after the tld part of the url, i.e. https://site.com./1235/article https://site.com./1235/article Those behind Cloudflare don't seem to be vulnerable to this though. I've emailed the sites I've found where this works and none of them have fixed it after a year.

neoromantique 4y ago
>I've emailed the sites I've found where this works Why?
- bilekas 4y ago
  Can't speak for OP but I think I would be annoyed if I was paying a subscription and I found out I could get around it with a simple `.` because of developer incompetence.
  - neoromantique 4y ago
    If you are paying for a subscription then hopefully you do it to support the publication and find their work useful, and not because of a nag banner.
    - bilekas 4y ago
      > to support the publication and find their work useful, and not because of a nag banner. Unfortunately they kind of go hand in hand these days!
    - z3t4 4y ago
      People don't pay for free stuff, even if they like it. People are however lazy, and many would pay to not have to put a dot in the URL every time.
  - manojlds 4y ago
    Good that you spoke for yourself.

suhaybh 4y ago

I just use https://github.com/iamadamdev/bypass-paywalls-chrome https://github.com/iamadamdev/bypass-paywalls-chrome and it works well for me.

alkonaut 4y ago

10 years ago paywalled sites contained the content just hidden. Today I haven't seen a site in a long time that renders the content hidden (why would it do that? There is no reason to do it based on indexing/SEO as far as I'm aware). Even cached/archived versions these days tend to not include the whole text. Basically: they figured out how to make a paywall, which frankly isn't that surprising.

donohoe 4y ago
Not sure I agree with that assessment. There are so many ways to do a paywall and you’ll still see all sorts of flavors across the web today.

jwr 4y ago

What I find annoying about paywalled sites is that they provide the full content to Google. And Google is OK with indexing the full content, even though it is not available on the internet, and even though they explicitly forbid the practice of showing different content to a search engine from what is available publicly. Paywalled sites are just fine, but they are not part of the open Internet, and should not pretend to be.

CM30 4y ago
Yeah, 100% agree with this. It's like these sites want to have their cake and eat it, and both get the traffic the 'open web' provides while not having to actually share any of their work there. It's like if you needed an app to view a page, yet Google had all its content indexed. Why is that (rightly) seen as unreasonable while charging users for content you provide to bots for free isn't?
donohoe 4y ago
> they explicitly forbid the > practice of showing different > content to a search engine from > what is available publicly This isn’t true. This paywall treatment is something they do allow and have worked to accommodate.
- tyingq 4y ago
  It's also true that they say "cloaking" isn't allowed, "any type of cloaking" https://youtu.be/QHtnfOgp65Q https://youtu.be/QHtnfOgp65Q
iicc 4y ago
Sometimes the full content is available - but only if you navigate to it via a Google results page, and you don't have existing cookies implementing a "free article limit".
mrobins 4y ago
That’s a Google problem not a publisher problem. Content should be findable and available to people who want to pay for it. Google could easily add an option to filter out paywalled content but that would reduce clicks.

snaix 4y ago

Shortcut in macos, in most browsers, to get into "div selection mode" is "shift + command + c"... Just press that, select the paywall (and any other junk backdrops/opaque divs), and press delete. Sometimes the site also sets an `overflow: hidden` in the css, and you need to remove that to see the content..

vixen99 4y ago

Did not work with https://www.spectator.co.uk/ https://www.spectator.co.uk/.

vixen99 4y ago

Does not work with https://www.spectator.co.uk/ https://www.spectator.co.uk/

phtrivier 4y ago

Title should be : "getting around very poorly implemented paywall (eg WaPo) with devtools alone". As soon as your site sends the whole content of the article to the browser, you're not even trying seriously. (And Firefox "reading mode" is just much better ux than the devtools.)

1vuio0pswjnm7 4y ago

Another option no one has menitoned is AMP. Many sites that try to use "paywalls" have AMP URLs which point to pages that have all the full text of the article in <p> tags. These AMP sites generally look great in a text-only browser that does not run Javascript. Popular example is WSJ. In the URL, add /amp before /article/. Paywalls are insidious because they target non-subscribers. Why let non-subscribers view articles. Why not password protect all subscriber content. Paywalls are a way to make money from (the attention of) non-subscribers, targeting them with ads and tracking. The strategy is apparently to annoy people to the point of subscribing. Yet even if they subscribe they will still be subjected to advertising. One potential advantage is that a paying subscriber has an enforceable contract. In theory the contract could contain enforceable privacy protections. "Tech" companies would never agree to give people enforceable privacy protections; it would destroy their "business". The way to save journalism, especially local news, is to regulate "Big Tech" middlemen, who generally do not employ journalists and produce zero content.1 The quality of journalism in general has taken a nosedive, but placing the blame for that on web users not purchasing subscriptions is conveniently ignoring the true culprit. 1. Arguably that's a prerequisite to maintaining their Section 230 protection. In the recent Supreme Court oral arguments, Google's counsel argued Google is not a publisher. Then minutes later she argued Google has to make design decisions "like any publisher", therefore Google gets a free pass to reorganise information in annoying and perhaps harmful ways to maximise ad services revenue, like inserting "popular" videos into YouTube search results that have nothing to do with the query string.

donohoe 4y ago
But the number of AMP versions is dwindling as Google is no longer forcing it.
- 1vuio0pswjnm7 4y ago
  Yep. And the browser vendor can make changes to DevTools, archive.ph is already privacy-unfriendly and blocked in some countries, it could disappear, 12ft.io could disappear, and so on. The fact that seems least likely to change is that most "paywalls" rely on Javascript, CSS or some other "feature" of so-called "modern" web browsers. I have lost count of how many times an HN comment alleges "paywalled" in a thread and I am reading the article just like any other because I am not using a graphical web browser. I would not even know there was a "paywall". Paywalls have dependencies.

porbelm 4y ago

Yeah on the system most of the local papers use over here, the full content is not even loaded for non-subscribers. It used to be, so removing the blur div worked, but now only the headline, byline and lead text are visible :( Guess they caught on to the "cheaters."

dspillett 4y ago

> But I didn't know how to scroll down on the page until today. That is usually due to an "overflow: hidden" somewhere near the top of the DOM tree. Remove that and your normal scrollbar usually returns. You see this a lot with "accept being stalked to read this" pop-overs as well as paywall related shenanigans. I've seen some sites put it back in via JS. There is probably a workaround for that too though the easiest one is to not worry about it and DNS blacklist the site so you don't waste time visiting it again in future.

Eupraxias 4y ago

Anyone else just read content in source?

jrberendt 4y ago

It's insane that news sites with thousands or even millions of monthly users have paywalls like this. Sure it's great for anyone who wants to read for free, but some individuals make their livings now through paywalled articles. If the big guys can't protect from exploits, what about everyone else? I created https://turbolink.io https://turbolink.io to attempt to solve this problem.

thallosaurus 4y ago

Oh yeah, used this method to bypass a regional paywalled news site until they fixed it by sending out scrambled text that seems random enough to not be a cipher ("Zc xjixc Axiäclxiqcil jcqlxi ljx Zxcxqjxiqxi cc")

iKlsR 4y ago

The simplest method is just to refresh the page and hit Esc as quickly as you can to cancel the page load and prevent further scripts from running, that works for me 80% of the time. Sites need to be indexed by google so any paywall is often client side js.

kokanee 4y ago

If you're unable to scroll the page, it generally means there is a "position: fixed" css rule on the body or a wrapper element. Turn off that css rule and you can scroll through the article normally. Publishers are slowly wising up, though. Most don't load the full article for unpaid viewers anymore.

pentagrama 4y ago

This extension removes paywalls on many news sites (maintained, 33k stars) https://github.com/iamadamdev/bypass-paywalls-chrome https://github.com/iamadamdev/bypass-paywalls-chrome

latchkey 4y ago
There was some previous drama and some people switched to this one. https://gitlab.com/magnolia1234/bypass-paywalls-chrome-clean/ https://gitlab.com/magnolia1234/bypass-paywalls-chrome-clean... https://gitlab.com/magnolia1234/bypass-paywalls-firefox-clean/-/issues/1 https://gitlab.com/magnolia1234/bypass-paywalls-firefox-clea...

oh_sigh 4y ago

Generally just disabling JavaScript for the site works perfectly.

l3mure 4y ago

Another devtools trick I've used is network throttling to allow me to copy out an article's content before JS loads.

71a54xd 4y ago

Your site's feature of "cool comments from HN" is awesome! Automating this with some kind of LLM and selling it would be killer.

Am4TIfIsER0ppos 4y ago

Just like with pointless loading spinners. You just delete the element which is overlaid on the complete content. Sometimes there is an overflow, opacity, or visibility attribute that needs changing. Fucking webdevs!

kleinmatic 4y ago

Pretty amazing how many people's ethics are less valuable to them than their product design opinions.

throwaway29303 4y ago

It seems the cat's out of the bag now. There's also another interesting way of getting around paywalls: a simple race condition. While loading the page hit stop as soon as you load enough of the content you're interested in et voilà. It does not work 100% of the time and I'm not using Chrome, however. ;)