6 ms·

Unlock Articles with Paywallskip

55 points by francocanzani 2y ago

francocanzani 2y ago
Hi! Our project addresses the limitations of existing paywall bypass tools by implementing a dynamic, community-driven approach. Key features include: Real-time Adaptive Blacklist: Constantly updated database of paywalled sites and effective bypass methods User-driven reporting system for quick adaptation to paywall changes Significantly faster response to new paywalls compared to static solutions Multi-Method Bypass Arsenal: Unlike single-method solutions (e.g., 12ft.io's cache access), we employ various techniques Methods include: User-Agent spoofing, Referer header manipulation, JS disabling post-load, and web archive fallbacks (Wayback Machine, archive.is, Google cache) Our blacklist determines the most effective method per site, improving success rates Site-Specific Solutions: Tracking individual websites allows for custom bypass methods when general approaches fail Parsed and validated HTML output ensures content integrity We believe this approach offers a more robust and adaptable solution to paywall bypassing. We're eager to hear the community's thoughts and potential improvements.
- Ancapistani 2y ago
  Is this open or shared source? Need help?
  - francocanzani 2y ago
    I'm really doubtful on this because most of these open source apps get banned from Github. Probably will make a dummy account to open issues and always have a copy local and in Gitlab. Should be able to clean the code a bit and share it soon.
    - bj-rn 2y ago
      Maybe put it on codeberg. https://codeberg.org https://codeberg.org https://en.wikipedia.org/wiki/Codeberg https://en.wikipedia.org/wiki/Codeberg
      - francocanzani 2y ago
        Will have a look. Thanks
    - Ancapistani 2y ago
      Even if you can’t host it on a web-based platform, a git repo accessible over SSH would be sufficient. I’m happy to submit patches via email :)
- foobarqux 2y ago
  Doesn’t seem to work on wsj, which is the only site archive.is doesn’t work on consistently anymore. So why not just use archive.is?
  - francocanzani 2y ago
    Will have a look. It worked earlier today.
  - ventegus 2y ago
    wsj has easy-to-bypass paywall but hard-to-bypass antibot/antiscraping solution, so the browser extensions work better than hosted bypassers
    - foobarqux 2y ago
      That bypass paywalls chrome extension has never worked for me on wsj, even on a fresh chromium install. What extension is supposed to work?
holistio 2y ago
This is going to be a shameless plug here. I'm also frustrated by how paywalls work, having to endlessly subscribe to yet another newspaper or newsletter, just because you want to read one article. But this approach is... stealing. When it comes to big media like the NYT, it might not feel like it is, but at the end of the day, it's people doing the background research, writing the content, editing, etc. This is _not_ the solution. Our alternative in the making offers a legal, fair alternative: buy and sell content by the piece. It's available at https://vernus.one https://vernus.one If you have a blog or newsletter, or if you're a journalist, I'm happy to help in setting up your publication and start earning in a transparent manner.
- m000 2y ago
  Can I get my money back if it turns out that some NYT article was outright lies, probably already paid for by some undisclosed shady entity?
  - holistio 2y ago
    I wish you could. I'd like Vernus to champion indie journalism, mainstream media is dying before our eyes anyway.
  - napoleoncomplex 2y ago
    Do you get your money back if you go to the store and buy some new food/fruit/snack you don't like the taste of? No, you throw it away, and probably won't buy it again. If you don't like NYT, don't buy from them. If NYT is like an avocado for you, sometimes ripe and delicious, sometimes unripe, sometimes rotten, you get to decide how often you're gonna buy avocados, or if you'll develop your own methods of avocado testing before buying to increase your odds. In no case do you get to take the avocado skin back to the store asking for a refund.
    - rafram 2y ago
      Trader Joe’s will give you your money back if you buy something and don’t like it.
rendall 2y ago
Does paywallskip scrape archive.is and archive.org?
- francocanzani 2y ago
  It fallbacks to archives, yes. Basically I use different User-Agent headers, different Referer headers, it tries disabling javascript once the page has loaded and the fallback is to fetch from web archives (Wayback Machine, archive.is, Google cache). Then the HTML is validated and parsed.
rrr_oh_man 2y ago
What will you do when the lawyers come for you?
- francocanzani 2y ago
  The legal landscape surrounding this issue remains ambiguous. I've documented my analysis in the legal section of my website. Typically, the consequence is domain takedowns, which is why I proactively purchased 10 domains as a precautionary measure. https://www.paywallskip.com/posts/legal https://www.paywallskip.com/posts/legal
  - hiergiltdiestfu 2y ago
    :D
  - aaomidi 2y ago
    Get a .is domain potentially. They kinda don’t really care about these takedown requests IIRC.
frankacter 2y ago
Are you considering a chrome extension to automate the process from the client perspective?
- francocanzani 2y ago
  Yes, is being developed and will launch this week. Just click and go, nothing fancy.
DataDaemon 2y ago
not every hero wears a cape
- upwardbound 2y ago
  "but all villains do" (This quip means that if a demagogue aggrandizes themself as a hero or savior, you should be suspicious of whether they really are as saintly as they claim to be.) Edit: I'm not saying this quip about paywallskip, which I think is a kindhearted service because it helps reduce the information gap, or digital divide, between the middle class and those stuck in poverty. Rather, the joke/warning is about demagogues in politics or in wealthy churches, stuff like that. Not about this post. Paywallskip never called themselves a hero - an appreciative user did, which is authentic and sweet. I think no one should ever call themselves a hero. Just help people, like Paywallskip is doing. I guess that what real heroes always say, both in movies and in real life, is "I was just doing my job". See e.g. this school bus driver who saved a child's life: https://www.youtube.com/shorts/xnW3NHbMTRs https://www.youtube.com/shorts/xnW3NHbMTRs (short, inspiring video)
emmet 2y ago
Not a fan of paywalls, but the framing of the pricing comparison is just so funny to me. Like the guy who comes into the pub offering to sell dubiously acquired meat for a bargain price.
brtkdotse 2y ago
I’d like something like this, but instead of bypassing the paywalls and stealing the articles it’d pay a few cents for each article and charge me at the end of the month. I really, really want to pay but I really, really don’t want to set up a subscription to read a one-off article.
- bluescrn 2y ago
  Nobody's every been prepared to try microtransactions that are actually micro. I'd like a browser addon that warns of paywalled links before I click them, perhaps adds a padlock icon or dollar sign (maybe with required subscription/cost on hover), to avoid nonsense like 'the first paragraph or two is free, then you hit the paywall'. Even better if it can do the same for region-locked sites and adblocker-blocking sites, or if on mobile, sites that nag/force you to use apps instead of the web...
- nelox 2y ago
  Blendle used to provide it, but alas, no more: https://en.m.wikipedia.org/wiki/Blendle https://en.m.wikipedia.org/wiki/Blendle
- salviati 2y ago
  I thought the Brave browser would provide such a solution. I was very disappointed to find out it's not an intended use case.
- yard2010 2y ago
  You are not stealing anything, unless you find a way to delete the source article after you downloaded it. It might break copyright rules. I wouldn't recommend doing either though.
  - lynx23 2y ago
    Hmm, I think the victims of organized corporate espionage might see this completely different. Its totally enough if "agents" of a certain country known for not really caring about copyright copy the plans for your new product. They dont have to delete yoour original plans to do damage.
truculent 2y ago
“Is using PaywallSkip legal?” is in the FAQs, but not “Is using PaywallSkip ethical?” Why do we feel so entitled to other people’s work as soon as it’s published on the internet? When Homer Simpson reads magazines at the convenience store without paying, we understand that he’s behaving poorly. Why do we feel differently now?
- deleted 2y ago
  [deleted]
- austinjp 2y ago
  Because the corner store has, in many cases, become a gigantic multinational monster that exploits the authors and the readers.
- mathgeek 2y ago
  > When Homer Simpson reads magazines at the convenience store without paying A more specific example would be ripping off the plastic packaging to read a magazine. The ones without extra packaging are designed and for a long time expected to be browsed.
- jampekka 2y ago
  > Why do we feel so entitled to other people’s work as soon as it’s published on the internet? Because intellectual property is a broken and harmful system.
- salviati 2y ago
  I tried to subscribe to an online newspaper in my country. Subscription was very easy. When I wanted to end it, they made it purposefully hard for me to do. I think it's unacceptable, and should be outlawed: the process to unsubscribe should be as hard (or easy) as the one to subscribe. Since newspapers (at least that specific one) have no problem doing this as long as it's legal, why should the public be more observant of ethics?
  - freefaler 2y ago
    If Revolut is available in your country, create a new virtual card for each subscription and when you can't/don't want to pay them anymore just cancel the card in Revolut.
nkrisc 2y ago
Do you also support skipping the paywall for services like AWS as well? That would be useful.
- francocanzani 2y ago
  That would take the fun out of AWS
master_crab 2y ago
Seems to be broken. Get a lot of server side error messages

55 points by francocanzani 2y ago

Hi! Our project addresses the limitations of existing paywall bypass tools by implementing a dynamic, community-driven approach. Key features include: Real-time Adaptive Blacklist: Constantly updated database of paywalled sites and effective bypass methods User-driven reporting system for quick adaptation to paywall changes Significantly faster response to new paywalls compared to static solutions Multi-Method Bypass Arsenal: Unlike single-method solutions (e.g., 12ft.io's cache access), we employ various techniques Methods include: User-Agent spoofing, Referer header manipulation, JS disabling post-load, and web archive fallbacks (Wayback Machine, archive.is, Google cache) Our blacklist determines the most effective method per site, improving success rates Site-Specific Solutions: Tracking individual websites allows for custom bypass methods when general approaches fail Parsed and validated HTML output ensures content integrity We believe this approach offers a more robust and adaptable solution to paywall bypassing. We're eager to hear the community's thoughts and potential improvements.

Ancapistani 2y ago
Is this open or shared source? Need help?
- francocanzani 2y ago
  I'm really doubtful on this because most of these open source apps get banned from Github. Probably will make a dummy account to open issues and always have a copy local and in Gitlab. Should be able to clean the code a bit and share it soon.
  - bj-rn 2y ago
    Maybe put it on codeberg. https://codeberg.org https://codeberg.org https://en.wikipedia.org/wiki/Codeberg https://en.wikipedia.org/wiki/Codeberg
    - francocanzani 2y ago
      Will have a look. Thanks
  - Ancapistani 2y ago
    Even if you can’t host it on a web-based platform, a git repo accessible over SSH would be sufficient. I’m happy to submit patches via email :)
foobarqux 2y ago
Doesn’t seem to work on wsj, which is the only site archive.is doesn’t work on consistently anymore. So why not just use archive.is?
- francocanzani 2y ago
  Will have a look. It worked earlier today.
- ventegus 2y ago
  wsj has easy-to-bypass paywall but hard-to-bypass antibot/antiscraping solution, so the browser extensions work better than hosted bypassers
  - foobarqux 2y ago
    That bypass paywalls chrome extension has never worked for me on wsj, even on a fresh chromium install. What extension is supposed to work?

holistio 2y ago

This is going to be a shameless plug here. I'm also frustrated by how paywalls work, having to endlessly subscribe to yet another newspaper or newsletter, just because you want to read one article. But this approach is... stealing. When it comes to big media like the NYT, it might not feel like it is, but at the end of the day, it's people doing the background research, writing the content, editing, etc. This is _not_ the solution. Our alternative in the making offers a legal, fair alternative: buy and sell content by the piece. It's available at https://vernus.one https://vernus.one If you have a blog or newsletter, or if you're a journalist, I'm happy to help in setting up your publication and start earning in a transparent manner.

m000 2y ago
Can I get my money back if it turns out that some NYT article was outright lies, probably already paid for by some undisclosed shady entity?
- holistio 2y ago
  I wish you could. I'd like Vernus to champion indie journalism, mainstream media is dying before our eyes anyway.
- napoleoncomplex 2y ago
  Do you get your money back if you go to the store and buy some new food/fruit/snack you don't like the taste of? No, you throw it away, and probably won't buy it again. If you don't like NYT, don't buy from them. If NYT is like an avocado for you, sometimes ripe and delicious, sometimes unripe, sometimes rotten, you get to decide how often you're gonna buy avocados, or if you'll develop your own methods of avocado testing before buying to increase your odds. In no case do you get to take the avocado skin back to the store asking for a refund.
  - rafram 2y ago
    Trader Joe’s will give you your money back if you buy something and don’t like it.

rendall 2y ago

Does paywallskip scrape archive.is and archive.org?

francocanzani 2y ago
It fallbacks to archives, yes. Basically I use different User-Agent headers, different Referer headers, it tries disabling javascript once the page has loaded and the fallback is to fetch from web archives (Wayback Machine, archive.is, Google cache). Then the HTML is validated and parsed.

rrr_oh_man 2y ago

What will you do when the lawyers come for you?

francocanzani 2y ago
The legal landscape surrounding this issue remains ambiguous. I've documented my analysis in the legal section of my website. Typically, the consequence is domain takedowns, which is why I proactively purchased 10 domains as a precautionary measure. https://www.paywallskip.com/posts/legal https://www.paywallskip.com/posts/legal
- hiergiltdiestfu 2y ago
  :D
- aaomidi 2y ago
  Get a .is domain potentially. They kinda don’t really care about these takedown requests IIRC.

frankacter 2y ago

Are you considering a chrome extension to automate the process from the client perspective?

francocanzani 2y ago
Yes, is being developed and will launch this week. Just click and go, nothing fancy.

DataDaemon 2y ago

not every hero wears a cape

upwardbound 2y ago
"but all villains do" (This quip means that if a demagogue aggrandizes themself as a hero or savior, you should be suspicious of whether they really are as saintly as they claim to be.) Edit: I'm not saying this quip about paywallskip, which I think is a kindhearted service because it helps reduce the information gap, or digital divide, between the middle class and those stuck in poverty. Rather, the joke/warning is about demagogues in politics or in wealthy churches, stuff like that. Not about this post. Paywallskip never called themselves a hero - an appreciative user did, which is authentic and sweet. I think no one should ever call themselves a hero. Just help people, like Paywallskip is doing. I guess that what real heroes always say, both in movies and in real life, is "I was just doing my job". See e.g. this school bus driver who saved a child's life: https://www.youtube.com/shorts/xnW3NHbMTRs https://www.youtube.com/shorts/xnW3NHbMTRs (short, inspiring video)

emmet 2y ago

Not a fan of paywalls, but the framing of the pricing comparison is just so funny to me. Like the guy who comes into the pub offering to sell dubiously acquired meat for a bargain price.

brtkdotse 2y ago

I’d like something like this, but instead of bypassing the paywalls and stealing the articles it’d pay a few cents for each article and charge me at the end of the month. I really, really want to pay but I really, really don’t want to set up a subscription to read a one-off article.

bluescrn 2y ago
Nobody's every been prepared to try microtransactions that are actually micro. I'd like a browser addon that warns of paywalled links before I click them, perhaps adds a padlock icon or dollar sign (maybe with required subscription/cost on hover), to avoid nonsense like 'the first paragraph or two is free, then you hit the paywall'. Even better if it can do the same for region-locked sites and adblocker-blocking sites, or if on mobile, sites that nag/force you to use apps instead of the web...
nelox 2y ago
Blendle used to provide it, but alas, no more: https://en.m.wikipedia.org/wiki/Blendle https://en.m.wikipedia.org/wiki/Blendle
salviati 2y ago
I thought the Brave browser would provide such a solution. I was very disappointed to find out it's not an intended use case.
yard2010 2y ago
You are not stealing anything, unless you find a way to delete the source article after you downloaded it. It might break copyright rules. I wouldn't recommend doing either though.
- lynx23 2y ago
  Hmm, I think the victims of organized corporate espionage might see this completely different. Its totally enough if "agents" of a certain country known for not really caring about copyright copy the plans for your new product. They dont have to delete yoour original plans to do damage.

truculent 2y ago

“Is using PaywallSkip legal?” is in the FAQs, but not “Is using PaywallSkip ethical?” Why do we feel so entitled to other people’s work as soon as it’s published on the internet? When Homer Simpson reads magazines at the convenience store without paying, we understand that he’s behaving poorly. Why do we feel differently now?

deleted 2y ago
[deleted]
austinjp 2y ago
Because the corner store has, in many cases, become a gigantic multinational monster that exploits the authors and the readers.
mathgeek 2y ago
> When Homer Simpson reads magazines at the convenience store without paying A more specific example would be ripping off the plastic packaging to read a magazine. The ones without extra packaging are designed and for a long time expected to be browsed.
jampekka 2y ago
> Why do we feel so entitled to other people’s work as soon as it’s published on the internet? Because intellectual property is a broken and harmful system.
salviati 2y ago
I tried to subscribe to an online newspaper in my country. Subscription was very easy. When I wanted to end it, they made it purposefully hard for me to do. I think it's unacceptable, and should be outlawed: the process to unsubscribe should be as hard (or easy) as the one to subscribe. Since newspapers (at least that specific one) have no problem doing this as long as it's legal, why should the public be more observant of ethics?
- freefaler 2y ago
  If Revolut is available in your country, create a new virtual card for each subscription and when you can't/don't want to pay them anymore just cancel the card in Revolut.

nkrisc 2y ago

Do you also support skipping the paywall for services like AWS as well? That would be useful.

francocanzani 2y ago
That would take the fun out of AWS

master_crab 2y ago

Seems to be broken. Get a lot of server side error messages