11 ms·

13ft – A site similar to 12ft.io but self-hosted

618 points by darknavi 2y ago

bansheeps 2y ago
I continue my search for a pay wall remover that will work with The Information. I'm honestly impressed that I've never been able to read an Information article in full.
- philistine 2y ago
  They probably never make their full articles available without a login. Google Search can probably index them well enough with just the excerpt. Simple as that.
  - csallen 2y ago
    There's definitely a hit to search traffic if you go this route, as Google is unlikely to rank you above a competing article for a competitive keyword based on only an excerpt. The Information simply doesn't care. They have an expensive subscription ($400/year) that I'd guess targets VCs and tech execs, which is a very specific segment that's best reached via means other than Google search, anyway. But yes, to your point, successfully paywalling a media site in a way that's impossible to bypass is trivially easy to do. Most media sites just don't think it's worth it.
darknavi 2y ago
I found this when looking for fun self hosted apps. It's pretty bare bones but does seem to work well with articles I've found so far.
j_maffe 2y ago
has 12ft.io even been working anymore? I feel like the only reliable way now is archive.is
- 91bananas 2y ago
  I just had anecdotal success with it last week and the atlantic, but before that it has been very hit and miss.
  - mvonballmo 2y ago
    I'm using the [Bypass Paywalls](https://github.com/iamadamdev/bypass-paywalls-chrome/blob/master/README.md https://github.com/iamadamdev/bypass-paywalls-chrome/blob/ma...) extension but it looks like that's been DMCA-ed in the interim.
    - Kikawala 2y ago
      https://github.com/bpc-clone/bypass-paywalls-firefox-clean https://github.com/bpc-clone/bypass-paywalls-firefox-clean or https://github.com/bpc-clone/bypass-paywalls-chrome-clean https://github.com/bpc-clone/bypass-paywalls-chrome-clean
      - thecal 2y ago
        Which directs you to https://gitflic.ru/project/magnolia1234/bpc_uploads https://gitflic.ru/project/magnolia1234/bpc_uploads which is not ideal...
        compuguy 2y ago
        I agree. Though there is a counterpoint that a Russian host isn't going to respect a DMCA request. On the flipside it's a Russian replacement for Github that is based on Gogs, Gitea, or even Forgejo possibly. So yeah, YMMV.
- Gurathnaka 2y ago
  Very rarely works.
- LetsGetTechnicl 2y ago
  Yeah I also have been using archive.today as well, since 12ft hasn't worked on NYT in forever
XCSme 2y ago
I am not familiar with 12ft.io, I wanted to try it out, but I get "Internal Server Error" when trying to visit a website.
refibrillator 2y ago
Running a server just to set the user agent header to the googlebot one for some requests feels a bit heavyweight. But perhaps it’s necessary, as it seems Firefox no longer has an about:config option to override the user agent…am I missing it somewhere? Edit: The about:config option general.useragent.override can be created and will be used for all requests (I just tested). I was confused because that config key doesn’t exist in a fresh install of Firefox. The user agent header string from this repo is: "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html http://www.google.com/bot.html)"
- darknavi 2y ago
  Personally I find it nice for sending articles to friends.
  - samstave 2y ago
    You can make a search related function in FF by rightclicking on that box and 'add keyword for this search' https://i.imgur.com/AkMxqIj.png https://i.imgur.com/AkMxqIj.png and then in your browser just type the letter you assign it to: for example, I have 'i' == the searchbox for IMDB, so I type 'i [movie]' in my url and it brings up the IMDB search of that movie . https://i.imgur.com/dXdwsbA.png https://i.imgur.com/dXdwsbA.png So you can just assign 'a' to that search box and type 'a [URL]' in your address bar and it will submit it to your little thing.
  - cortesoft 2y ago
    That would mean that your self-hosted install is exposed to the internet. I don't think I want to run a publicly accessible global relay.
    - KennyBlanken 2y ago
      Eh, pretty minimal risk unless you use a guessable hostname and/or the URL gets published somewhere. If the install is under "eixk3.somedomain.com/ladderforfriends" and it sits behind a reverse proxy, it might as well be invisible to the internet, unless your DNS provider is an idiot and allows zone transfers, or you are on a network where someone is snarfing up DNS requests and then distributing that info to third parties. If you restrict it to TLS 1.3, even someone sniffing traffic from one of your friends won't be able to glean anything useful, because the requested hostname is never sent in plaintext. Rotate the hostname if/when it becomes necessary...
xyst 2y ago
I’m more inclined to use archive(.org|.ph). But this is a decent workaround when archive is unavailable. Side note: paywalls are annoying but most publications are often available for free via public library. For example, NYT is free via my public library. PL offers 3-day subs. A few other decent publications are available as well. Availability of publications is YMMV as well.
- pmdr 2y ago
  NYT's onion site is also free.
  - xyst 2y ago
    Onion site? As in, there’s a mirror on tor? (Edit) TIL: https://open.nytimes.com/https-open-nytimes-com-the-new-york-times-as-a-tor-onion-service-e0d0b67b7482 https://open.nytimes.com/https-open-nytimes-com-the-new-york...
- latexr 2y ago
  > most publications are often available for free via public library. Via public library in the USA. Other countries exist and as far as I’ve gathered aren’t typically given this level of free access.
  - xyst 2y ago
    Hence “ymmv” (your mileage may vary) ;)
    - elashri 2y ago
      Ironically this sentence origin is in the US context. And the abbreviation is mostly used in American English slang [1]. Please don't take it as attack or even criticism, I just found it funny observation. That might be wrong [1] https://en.wiktionary.org/wiki/your_mileage_may_vary https://en.wiktionary.org/wiki/your_mileage_may_vary
  - prmoustache 2y ago
mattbillenstein 2y ago
Counterpoint - if you like the content enough to go through this - just pay for it. Monetary support of journalism or content you like is a great way to encourage more of it.
- ed 2y ago
  I fully agree with the sentiment! I support and do pay for sources I read frequently. Sadly payment models are incompatible with how most people consume content – which is to read a small number of articles from a large number of sources.
- naltroc 2y ago
  as much as I circumvent paywalls myself, it does feel like overkill to setup software to do it always. Sites spend money to produce quality content. Somewhat related comparison, Is a human choosing to do this theft really better than a neural network scraping content for its own purposes?
  - pjot 2y ago
    > Is a human choosing to do this theft really better than a neural network scraping content Probably so. I think the differentiation is in the scale at which scraping is done for those kinds of systems.
    - janalsncm 2y ago
      It’s probably about the same. The difference with sites like e.g. Perplexity is that they have a business model which requires “acquiring” said content for free whereas a single person is just a single person.
  - fwip 2y ago
    The neural network is not scraping content for its own purposes, it is for the purpose of the people who are running/training it. And yes, one person reading a piece of content without paying money for it is far, far better than one person/corporation scraping all of the world's content in order to profit off of it.
mgiampapa 2y ago
Bypass Paywalls Clean has moved here btw, https://github.com/bpc-clone?tab=repositories https://github.com/bpc-clone?tab=repositories
- pogue 2y ago
  BPC also has the option to spoof the User-agent as well when using the "custom sites" option: • set useragent to Googlebot, Bingbot, Facebookbot or custom • set referer (to Facebook, Google, Twitter or custom; ignored when Googlebot is set)
- compuguy 2y ago
  Yes, but it may be taken down via DMCA soon. See this DMCA request: https://github.com/github/dmca/blob/master/2024/08/2024-08-09-news-media-alliance.md https://github.com/github/dmca/blob/master/2024/08/2024-08-0... It mentions bpc_updates in the takedown request....
- compuguy 2y ago
  It moved to a Russian equivalent of GitHub or Gitlab (because of where its hosted, YMMV): https://gitflic.ru/project/magnolia1234/bpc_uploads https://gitflic.ru/project/magnolia1234/bpc_uploads
  - mgiampapa 2y ago
    That's not the URL posted on magnolia1234's Twitter, but it may be a mirror. Caveat emptor. I would watch here in case the primary goes down. https://x.com/Magnolia1234B https://x.com/Magnolia1234B
    - trissi1996 2y ago
      You're wrong, it's actually the exact URL posted on twitter: https://x.com/Magnolia1234B/status/1823073077867287028 https://x.com/Magnolia1234B/status/1823073077867287028
      - 2y ago
karmakaze 2y ago
Missed opportunity to call it 2ft, as in standing on one's own.
- trackofalljades 2y ago
  ...or 11ft8, which can open anything
  - spoonfeeder006 2y ago
    666ft....
  - numpad0 2y ago
    https://en.wikipedia.org/wiki/11foot8
    - karmakaze 2y ago
      I like how google shows "How tall is the 11-foot-8 bridge?" "12 feet 4 inches" (because maintenance work was done on it, but retaining its name)
- dredmorbius 2y ago
  I kind of like the implied concept that self-hosting is the last-foot problem.
sam_goody 2y ago
It seems to me that google should not allow a site to serve different content to their bot than they serve to their users. If the content is unavailable to me, it should not be in the search results. It obviously doesn't seem that way to Google, or to the sites providing the content. They are doing what works for them without ethical constraints (Google definitely, many content providers, eg NYT). Is it fair game to do what works for you (eg. 13ft)?!
- rurp 2y ago
  > It seems to me that google should not allow a site to serve different content to their bot than they serve to their users. That would be the fair thing to do and was Google's policy for many years, and still is for all I know. But modern Google stopped caring about fairness and similar concerns many years ago.
  - sltkr 2y ago
    The policy was that if a user lands on the page from the Google search results page, then they should be shown the full content, same as Googlebot (“First Click Free”). But that policy was abandoned in 2017: https://www.theguardian.com/technology/2017/oct/02/google-to-ditch-controversial-first-click-free-policy https://www.theguardian.com/technology/2017/oct/02/google-to...
- efilife 2y ago
  This is called cloaking[0] and is against Google's policies for many years. But they don't care [0] https://en.wikipedia.org/wiki/Cloaking https://en.wikipedia.org/wiki/Cloaking
- justinl33 2y ago
  "organizing the world's information"
- querez 2y ago
  I disagree, I think the current approach actually makes for a better and more open web long term. The fact is that either you pay for content, or the content has to pay for itself (which means it's either sponsored content or full of ads). Real journalism costs money, there's no way around that. So we're left with a few options: Option a) NYT and other news sites makes their news open to everyone without paywall. To finance itself it will become full of ads and disgusting. Option b) NYT and other news sites become fully walled gardens, letting no-one in (including Google bots). It won't be indexed by Google and search sites, we won't be able to find its context freely. It's like a discord site or facebook groups: there's a treasure trove of information out there, but you won't be able to find it when you need it. Option c): NYT and other news sites let Google and search sites index their content, but asks the user to pay to access it.
spoonfeeder006 2y ago
Why not just use uBlock Origin for the aspect of cleaning up the popups / ads and such?
- brikym 2y ago
  Because I was stupid and bought an iPhone which means that is not possible.
  - TimAppleman 2y ago
    Do iPhones not have Firefox now?
    - nzgrover 2y ago
      Quoting _khhm who posted here: https://news.ycombinator.com/item?id=25850091 https://news.ycombinator.com/item?id=25850091 > On iOS there are no web browsers other than Safari, per the app store rules. "Chrome" / "Firefox" / etc on iOS are just basically skins on top of Webkit. > See 2.5.6 here - https://developer.apple.com/app-store/review/guidelines/ https://developer.apple.com/app-store/review/guidelines/ > This is why you don't get any of the features / extensions / etc of Chrome or Firefox on iOS.
      - johnisgood 2y ago
        > per the app store rules Jeez.
      - bwag 2y ago
        With iOS 17.4 they allow other rendering engines for users in the EU. https://www.apple.com/newsroom/2024/01/apple-announces-changes-to-ios-safari-and-the-app-store-in-the-european-union/ https://www.apple.com/newsroom/2024/01/apple-announces-chang...
  - ycombinete 2y ago
    AdGuard works on ios, so does 1blocker. They’re no uBlock, but they do the trick.
declan_roberts 2y ago
12ft.io doesn't really work anymore. If you're on iOS + Safari I recommend the "open in internet archive" shortcut, which is actually able to bypass most paywalls. https://www.reddit.com/r/shortcuts/comments/12fbk8m/ive_created_the_ultimate_shortcut_for_finding/ https://www.reddit.com/r/shortcuts/comments/12fbk8m/ive_crea...
Zambyte 2y ago
This is awesome! People who use Kagi can also set up a regex redirect to automatically use this for problematic sites.
linsomniac 2y ago
I'll gladly pay for journalist content, but not when a single article is going to be $15/mo and hard to cancel. Is there some way to support journalism across publications?
- deleted 2y ago
  [deleted]
- warkdarrior 2y ago
  Articles should come with license agreements, just like open source software nowadays. Free for personal entertainment, but if you try to make money from the information in the article or otherwise commercialize it, you can fuck right off.
  - throw10920 2y ago
    > Free for personal entertainment Didn't the GP say > Is there some way to support journalism across publications? I don't think there's a way to support without paying.
  - CaptainFever 2y ago
    > Free for personal entertainment, but if you try to make money from the information in the article or otherwise commercialize it, you can fuck right off. Note that such a license would not be considered open source. Open source and free software allows commercialization because they do not allow discrimination against various interest groups. The only thing that open source allows you to do is to restrict people from restricting the information, which has some relation to commercialization, but not fully.
- Marsymars 2y ago
  > Is there some way to support journalism across publications? Apple News+, Inkl, PressReader, (maybe more). Others if you want more magazine-focused subscriptions.
  - shanecleveland 2y ago
deskr 2y ago
It once was Google's requirement that you'd serve the same content to the Google crawler as to any other user. No surprise that Google is full of shit these days.
- justinl33 2y ago
  "organizing the world's information" or 'maximizing revenue?' I don't know - somehow either argument justifies this
- deleted 2y ago
  [deleted]
- deleted 2y ago
  [deleted]
BLKNSLVR 2y ago
This could be used as a proxy to web interfaces on the same local network couldn't it? There are probably much better and more secure options, but this might be an interesting temporary kludge.
ThinkBeat 2y ago
Does it help when pretending to the google bot to be running on an IP from inside the Google Cloud?
hammock 2y ago
Now if someone could just package this into a browser extension it would be great!
lutusp 2y ago
Nice effort, but after one successful NYT session, it fails and treats the access as though it were an end user. But don't take my word for it : try it. One access, succeeds. Two or more ... fails. The reason is the staff at the NYT appear to be very well versed in the technical tricks people use to gain access.
- echelon 2y ago
  We need a P2P swarm for content. Just like Bittorrent back in the day. Pop in your favorite news article (or paraphrase it yourself), and everyone gets it. With recommender systems, attention graph modeling, etc. it'd probably be a perfect information ingestion and curation engine. And nobody else could modify the algorithm on my behalf.
  - Swarmeggcute 2y ago
    I get the feeling there is some sort of vector of attack behind this idea, but I'm not well versed enough to figure it out.
  - deleted 2y ago
    [deleted]
  - bredren 2y ago
    I think you have something here. I don’t know about paraphrased versions but it would need to handle content revisions by the publisher somehow.
- WatchDog 2y ago
  They probably asynchronously verify that the IP address actually belongs to googlebot, then ban the IP when it fails. Synchronously verifying it, would probably be too slow. You can verify googlebot authenticity by doing a reverse dns lookup, then checking that reverse dns name resolves correctly to the expected IP address[0]. [0]: https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot https://developers.google.com/search/docs/crawling-indexing/...
selcuka 2y ago
I don't think this will work reliably as other commenters pointed out. A better solution could be to pass the URL through an archiver, such as archive.today: https://archive.is/20240719082825/https://www.nytimes.com/2024/07/18/us/frontier-flights-grounded-microsoft.html https://archive.is/20240719082825/https://www.nytimes.com/20...
- thinkloop 2y ago
  How do they do it?
bruce511 2y ago
>> This is a simple self hosted server that has a simple but powerful interface to block ads, paywalls, and other nonsense. Specially for sites like medium, new york times which have paid articles that you normally cannot read. Now I do want you to support the creators you benefit from but if you just wanna see one single article and move on with your day then this might be helpful Personally I'm not a fan of this attitude. I've read and digested the arguments for it, but, for me, it runs close to "theft". For example, read the sentence again, but in the context of a restaurant. Sure I wanna support the creators, but what if I just want a single meal and then get on with my day? Businesses, including news web sites, need to monetize their content. There are a variety of ways they do that. You are free to consume their content or not. You either accept their monetization method as desirable or you do not. The "I just want to read one article" argument doesn't fly. If the article is so compelling, then follow their rules for accessing it. Yes, some sites behave badly. So stop visiting them. There is lots of free content on the web that is well presented and lacks corporate malfeasance. Read some of that instead. I get that I'm gonna get downvoted to oblivion with this post. HN readers are in love with ad blockers and paywall bypasses. But just because you can do something, just because you think it should be "free, no ads", does not make it right. Creators create. They get to choose how the world sees their creation. Support it, don't support it, that's up to you. Deciding to just take it anyway, on your terms (however righteous you feel you are) is not ok.
- Daneel_ 2y ago
  I agree with you. Where the gap lies for me is that I can’t just buy one meal, I have to sign up for the yearly meal plan even if I just want one meal. A few years ago I tracked how many times I visited some of the various paywalls sites for their articles, and typically it was between 5-10 times per year. One was 30, so I paid for a subscription to them, but I can’t justify several dozens of dollars for 5 articles on many other sites. If I can’t access their content because a bypass doesn’t work then so be it, however I wasn’t willing to pay for that content either. I feel like it’s the classic misconception regarding piracy by the movie industry - I wasn’t willing to pay money for it in the first place, so it’s not lost revenue (unfortunately). I was actually discussing this overall problem with my wife the other day, and I came to the conclusion that I basically want a “Netflix” but for news - I subscribe to one place and I get access to a whole range of publications. That’s worth it to me. I very much don’t see it happening though, sadly.
Animats 2y ago
The next step being 11ft 8 inches.[1] [1] http://11foot8.com/ http://11foot8.com/
- willhackett 2y ago
  Reminds me of Montague Street Bridge. [1] [1] https://howmanydayssincemontaguestreetbridgehasbeenhit.com/ https://howmanydayssincemontaguestreetbridgehasbeenhit.com/
- Jenk 2y ago
  > The next step Should that read "Previous"? :)
- mvc 2y ago
  14ft. Similar to 13ft but implemented in Rust.
  - auadix 2y ago
    Or 4.2672m, for the rest of the world. :)
- deleted 2y ago
  [deleted]
drowntoge 2y ago
> Port 5000 is in use by another program. Either identify and stop that program, or start the server with a different port. An instruction on how to specify port would be nice.
- RoseyWasTaken 2y ago
  The docker-compose.yaml file is where you specify the ports you want to expose. It looks like by default it's 5000:5000 (5000 outside and inside the container). You will need to change it and then run docker-compose up -d. You can change it to something like 5133:5000 and access the instance through localhost:5133
  - drowntoge 2y ago
    Thank you for the tip! I ended up editing the port parameter in the app.run() call within portable.py and it worked. Felt like it might be a good idea to add this as a runtime argument for easier customization.
Ikatza 2y ago
It's always seemed easier to me to use FF + Ublock + Bypass Paywalls. Never fails.
wasi_master 2y ago
Hello everyone, it's the author here. I initially created 13ft as a proof of concept, simply to test whether the idea would work. I never anticipated it would gain this much traction or become as popular as it has. I'm thrilled that so many of you have found it useful, and I'm truly grateful for all the support. Regarding the limitations of this approach, I'm fully aware that it isn't perfect, and it was never intended to be. It was just a quick experiment to see if the concept was feasible—and it seems that, at least sometimes, it is. Thank you all for the continued support.
- darknavi 2y ago
  Apologies for submitting it here if it caused any sense of being overwhelmed. Hopefully FOSS is supportive here instead of overwhelming. Thanks for sharing the project with the internet!
ck2 2y ago
Just a caution you are probably headed for this result if hosted on GitHub (or GitLab) https://github.com/bpc-clone/bypass-paywalls-clean-filters https://github.com/bpc-clone/bypass-paywalls-clean-filters https://news.ycombinator.com/item?id=40007670 https://news.ycombinator.com/item?id=40007670 https://news.ycombinator.com/item?id=40015961 https://news.ycombinator.com/item?id=40015961
efangs 2y ago
From my experience, pihole is very easy to setup for this use case: https://pi-hole.net/ https://pi-hole.net/
- deleted 2y ago
  [deleted]
martin82 2y ago
I'm surprised this works at all. What sysadmin is so naive to rely on the Googlebot useragent?? Doesn't everyone fetch the official list of Googlebot IPs and then add those to a whitelist?

618 points by darknavi 2y ago

I continue my search for a pay wall remover that will work with The Information. I'm honestly impressed that I've never been able to read an Information article in full.

philistine 2y ago
They probably never make their full articles available without a login. Google Search can probably index them well enough with just the excerpt. Simple as that.
- csallen 2y ago
  There's definitely a hit to search traffic if you go this route, as Google is unlikely to rank you above a competing article for a competitive keyword based on only an excerpt. The Information simply doesn't care. They have an expensive subscription ($400/year) that I'd guess targets VCs and tech execs, which is a very specific segment that's best reached via means other than Google search, anyway. But yes, to your point, successfully paywalling a media site in a way that's impossible to bypass is trivially easy to do. Most media sites just don't think it's worth it.

darknavi 2y ago

I found this when looking for fun self hosted apps. It's pretty bare bones but does seem to work well with articles I've found so far.

j_maffe 2y ago

has 12ft.io even been working anymore? I feel like the only reliable way now is archive.is

91bananas 2y ago
I just had anecdotal success with it last week and the atlantic, but before that it has been very hit and miss.
- mvonballmo 2y ago
  I'm using the [Bypass Paywalls](https://github.com/iamadamdev/bypass-paywalls-chrome/blob/master/README.md https://github.com/iamadamdev/bypass-paywalls-chrome/blob/ma...) extension but it looks like that's been DMCA-ed in the interim.
  - Kikawala 2y ago
    https://github.com/bpc-clone/bypass-paywalls-firefox-clean https://github.com/bpc-clone/bypass-paywalls-firefox-clean or https://github.com/bpc-clone/bypass-paywalls-chrome-clean https://github.com/bpc-clone/bypass-paywalls-chrome-clean
    - thecal 2y ago
      Which directs you to https://gitflic.ru/project/magnolia1234/bpc_uploads https://gitflic.ru/project/magnolia1234/bpc_uploads which is not ideal...
      - compuguy 2y ago
        I agree. Though there is a counterpoint that a Russian host isn't going to respect a DMCA request. On the flipside it's a Russian replacement for Github that is based on Gogs, Gitea, or even Forgejo possibly. So yeah, YMMV.
Gurathnaka 2y ago
Very rarely works.
LetsGetTechnicl 2y ago
Yeah I also have been using archive.today as well, since 12ft hasn't worked on NYT in forever

XCSme 2y ago

I am not familiar with 12ft.io, I wanted to try it out, but I get "Internal Server Error" when trying to visit a website.

refibrillator 2y ago

Running a server just to set the user agent header to the googlebot one for some requests feels a bit heavyweight. But perhaps it’s necessary, as it seems Firefox no longer has an about:config option to override the user agent…am I missing it somewhere? Edit: The about:config option general.useragent.override can be created and will be used for all requests (I just tested). I was confused because that config key doesn’t exist in a fresh install of Firefox. The user agent header string from this repo is: "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html http://www.google.com/bot.html)"

darknavi 2y ago
Personally I find it nice for sending articles to friends.
- samstave 2y ago
  You can make a search related function in FF by rightclicking on that box and 'add keyword for this search' https://i.imgur.com/AkMxqIj.png https://i.imgur.com/AkMxqIj.png and then in your browser just type the letter you assign it to: for example, I have 'i' == the searchbox for IMDB, so I type 'i [movie]' in my url and it brings up the IMDB search of that movie . https://i.imgur.com/dXdwsbA.png https://i.imgur.com/dXdwsbA.png So you can just assign 'a' to that search box and type 'a [URL]' in your address bar and it will submit it to your little thing.
- cortesoft 2y ago
  That would mean that your self-hosted install is exposed to the internet. I don't think I want to run a publicly accessible global relay.
  - KennyBlanken 2y ago
    Eh, pretty minimal risk unless you use a guessable hostname and/or the URL gets published somewhere. If the install is under "eixk3.somedomain.com/ladderforfriends" and it sits behind a reverse proxy, it might as well be invisible to the internet, unless your DNS provider is an idiot and allows zone transfers, or you are on a network where someone is snarfing up DNS requests and then distributing that info to third parties. If you restrict it to TLS 1.3, even someone sniffing traffic from one of your friends won't be able to glean anything useful, because the requested hostname is never sent in plaintext. Rotate the hostname if/when it becomes necessary...

xyst 2y ago

I’m more inclined to use archive(.org|.ph). But this is a decent workaround when archive is unavailable. Side note: paywalls are annoying but most publications are often available for free via public library. For example, NYT is free via my public library. PL offers 3-day subs. A few other decent publications are available as well. Availability of publications is YMMV as well.

pmdr 2y ago
NYT's onion site is also free.
- xyst 2y ago
  Onion site? As in, there’s a mirror on tor? (Edit) TIL: https://open.nytimes.com/https-open-nytimes-com-the-new-york-times-as-a-tor-onion-service-e0d0b67b7482 https://open.nytimes.com/https-open-nytimes-com-the-new-york...
latexr 2y ago
> most publications are often available for free via public library. Via public library in the USA. Other countries exist and as far as I’ve gathered aren’t typically given this level of free access.
- xyst 2y ago
  Hence “ymmv” (your mileage may vary) ;)
  - elashri 2y ago
    Ironically this sentence origin is in the US context. And the abbreviation is mostly used in American English slang [1]. Please don't take it as attack or even criticism, I just found it funny observation. That might be wrong [1] https://en.wiktionary.org/wiki/your_mileage_may_vary https://en.wiktionary.org/wiki/your_mileage_may_vary
- prmoustache 2y ago

mattbillenstein 2y ago

Counterpoint - if you like the content enough to go through this - just pay for it. Monetary support of journalism or content you like is a great way to encourage more of it.

ed 2y ago
I fully agree with the sentiment! I support and do pay for sources I read frequently. Sadly payment models are incompatible with how most people consume content – which is to read a small number of articles from a large number of sources.
naltroc 2y ago
as much as I circumvent paywalls myself, it does feel like overkill to setup software to do it always. Sites spend money to produce quality content. Somewhat related comparison, Is a human choosing to do this theft really better than a neural network scraping content for its own purposes?
- pjot 2y ago
  > Is a human choosing to do this theft really better than a neural network scraping content Probably so. I think the differentiation is in the scale at which scraping is done for those kinds of systems.
  - janalsncm 2y ago
    It’s probably about the same. The difference with sites like e.g. Perplexity is that they have a business model which requires “acquiring” said content for free whereas a single person is just a single person.
- fwip 2y ago
  The neural network is not scraping content for its own purposes, it is for the purpose of the people who are running/training it. And yes, one person reading a piece of content without paying money for it is far, far better than one person/corporation scraping all of the world's content in order to profit off of it.

mgiampapa 2y ago

Bypass Paywalls Clean has moved here btw, https://github.com/bpc-clone?tab=repositories https://github.com/bpc-clone?tab=repositories

pogue 2y ago
BPC also has the option to spoof the User-agent as well when using the "custom sites" option: • set useragent to Googlebot, Bingbot, Facebookbot or custom • set referer (to Facebook, Google, Twitter or custom; ignored when Googlebot is set)
compuguy 2y ago
Yes, but it may be taken down via DMCA soon. See this DMCA request: https://github.com/github/dmca/blob/master/2024/08/2024-08-09-news-media-alliance.md https://github.com/github/dmca/blob/master/2024/08/2024-08-0... It mentions bpc_updates in the takedown request....
compuguy 2y ago
It moved to a Russian equivalent of GitHub or Gitlab (because of where its hosted, YMMV): https://gitflic.ru/project/magnolia1234/bpc_uploads https://gitflic.ru/project/magnolia1234/bpc_uploads
- mgiampapa 2y ago
  That's not the URL posted on magnolia1234's Twitter, but it may be a mirror. Caveat emptor. I would watch here in case the primary goes down. https://x.com/Magnolia1234B https://x.com/Magnolia1234B
  - trissi1996 2y ago
    You're wrong, it's actually the exact URL posted on twitter: https://x.com/Magnolia1234B/status/1823073077867287028 https://x.com/Magnolia1234B/status/1823073077867287028
    - 2y ago

karmakaze 2y ago

Missed opportunity to call it 2ft, as in standing on one's own.

trackofalljades 2y ago
...or 11ft8, which can open anything
- spoonfeeder006 2y ago
  666ft....
- numpad0 2y ago
  https://en.wikipedia.org/wiki/11foot8
  - karmakaze 2y ago
    I like how google shows "How tall is the 11-foot-8 bridge?" "12 feet 4 inches" (because maintenance work was done on it, but retaining its name)
dredmorbius 2y ago
I kind of like the implied concept that self-hosting is the last-foot problem.

sam_goody 2y ago

It seems to me that google should not allow a site to serve different content to their bot than they serve to their users. If the content is unavailable to me, it should not be in the search results. It obviously doesn't seem that way to Google, or to the sites providing the content. They are doing what works for them without ethical constraints (Google definitely, many content providers, eg NYT). Is it fair game to do what works for you (eg. 13ft)?!

rurp 2y ago
> It seems to me that google should not allow a site to serve different content to their bot than they serve to their users. That would be the fair thing to do and was Google's policy for many years, and still is for all I know. But modern Google stopped caring about fairness and similar concerns many years ago.
- sltkr 2y ago
  The policy was that if a user lands on the page from the Google search results page, then they should be shown the full content, same as Googlebot (“First Click Free”). But that policy was abandoned in 2017: https://www.theguardian.com/technology/2017/oct/02/google-to-ditch-controversial-first-click-free-policy https://www.theguardian.com/technology/2017/oct/02/google-to...
efilife 2y ago
This is called cloaking[0] and is against Google's policies for many years. But they don't care [0] https://en.wikipedia.org/wiki/Cloaking https://en.wikipedia.org/wiki/Cloaking
justinl33 2y ago
"organizing the world's information"
querez 2y ago
I disagree, I think the current approach actually makes for a better and more open web long term. The fact is that either you pay for content, or the content has to pay for itself (which means it's either sponsored content or full of ads). Real journalism costs money, there's no way around that. So we're left with a few options: Option a) NYT and other news sites makes their news open to everyone without paywall. To finance itself it will become full of ads and disgusting. Option b) NYT and other news sites become fully walled gardens, letting no-one in (including Google bots). It won't be indexed by Google and search sites, we won't be able to find its context freely. It's like a discord site or facebook groups: there's a treasure trove of information out there, but you won't be able to find it when you need it. Option c): NYT and other news sites let Google and search sites index their content, but asks the user to pay to access it.

spoonfeeder006 2y ago

Why not just use uBlock Origin for the aspect of cleaning up the popups / ads and such?

brikym 2y ago
Because I was stupid and bought an iPhone which means that is not possible.
- TimAppleman 2y ago
  Do iPhones not have Firefox now?
  - nzgrover 2y ago
    Quoting _khhm who posted here: https://news.ycombinator.com/item?id=25850091 https://news.ycombinator.com/item?id=25850091 > On iOS there are no web browsers other than Safari, per the app store rules. "Chrome" / "Firefox" / etc on iOS are just basically skins on top of Webkit. > See 2.5.6 here - https://developer.apple.com/app-store/review/guidelines/ https://developer.apple.com/app-store/review/guidelines/ > This is why you don't get any of the features / extensions / etc of Chrome or Firefox on iOS.
    - johnisgood 2y ago
      > per the app store rules Jeez.
    - bwag 2y ago
      With iOS 17.4 they allow other rendering engines for users in the EU. https://www.apple.com/newsroom/2024/01/apple-announces-changes-to-ios-safari-and-the-app-store-in-the-european-union/ https://www.apple.com/newsroom/2024/01/apple-announces-chang...
- ycombinete 2y ago
  AdGuard works on ios, so does 1blocker. They’re no uBlock, but they do the trick.

declan_roberts 2y ago

12ft.io doesn't really work anymore. If you're on iOS + Safari I recommend the "open in internet archive" shortcut, which is actually able to bypass most paywalls. https://www.reddit.com/r/shortcuts/comments/12fbk8m/ive_created_the_ultimate_shortcut_for_finding/ https://www.reddit.com/r/shortcuts/comments/12fbk8m/ive_crea...

Zambyte 2y ago

This is awesome! People who use Kagi can also set up a regex redirect to automatically use this for problematic sites.

linsomniac 2y ago

I'll gladly pay for journalist content, but not when a single article is going to be $15/mo and hard to cancel. Is there some way to support journalism across publications?

deleted 2y ago
[deleted]
warkdarrior 2y ago
Articles should come with license agreements, just like open source software nowadays. Free for personal entertainment, but if you try to make money from the information in the article or otherwise commercialize it, you can fuck right off.
- throw10920 2y ago
  > Free for personal entertainment Didn't the GP say > Is there some way to support journalism across publications? I don't think there's a way to support without paying.
- CaptainFever 2y ago
  > Free for personal entertainment, but if you try to make money from the information in the article or otherwise commercialize it, you can fuck right off. Note that such a license would not be considered open source. Open source and free software allows commercialization because they do not allow discrimination against various interest groups. The only thing that open source allows you to do is to restrict people from restricting the information, which has some relation to commercialization, but not fully.
Marsymars 2y ago
> Is there some way to support journalism across publications? Apple News+, Inkl, PressReader, (maybe more). Others if you want more magazine-focused subscriptions.
- shanecleveland 2y ago

deskr 2y ago

It once was Google's requirement that you'd serve the same content to the Google crawler as to any other user. No surprise that Google is full of shit these days.

justinl33 2y ago
"organizing the world's information" or 'maximizing revenue?' I don't know - somehow either argument justifies this
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]

BLKNSLVR 2y ago

This could be used as a proxy to web interfaces on the same local network couldn't it? There are probably much better and more secure options, but this might be an interesting temporary kludge.

ThinkBeat 2y ago

Does it help when pretending to the google bot to be running on an IP from inside the Google Cloud?

hammock 2y ago

Now if someone could just package this into a browser extension it would be great!

lutusp 2y ago

Nice effort, but after one successful NYT session, it fails and treats the access as though it were an end user. But don't take my word for it : try it. One access, succeeds. Two or more ... fails. The reason is the staff at the NYT appear to be very well versed in the technical tricks people use to gain access.

echelon 2y ago
We need a P2P swarm for content. Just like Bittorrent back in the day. Pop in your favorite news article (or paraphrase it yourself), and everyone gets it. With recommender systems, attention graph modeling, etc. it'd probably be a perfect information ingestion and curation engine. And nobody else could modify the algorithm on my behalf.
- Swarmeggcute 2y ago
  I get the feeling there is some sort of vector of attack behind this idea, but I'm not well versed enough to figure it out.
- deleted 2y ago
  [deleted]
- bredren 2y ago
  I think you have something here. I don’t know about paraphrased versions but it would need to handle content revisions by the publisher somehow.
WatchDog 2y ago
They probably asynchronously verify that the IP address actually belongs to googlebot, then ban the IP when it fails. Synchronously verifying it, would probably be too slow. You can verify googlebot authenticity by doing a reverse dns lookup, then checking that reverse dns name resolves correctly to the expected IP address[0]. [0]: https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot https://developers.google.com/search/docs/crawling-indexing/...

selcuka 2y ago

I don't think this will work reliably as other commenters pointed out. A better solution could be to pass the URL through an archiver, such as archive.today: https://archive.is/20240719082825/https://www.nytimes.com/2024/07/18/us/frontier-flights-grounded-microsoft.html https://archive.is/20240719082825/https://www.nytimes.com/20...

thinkloop 2y ago
How do they do it?

bruce511 2y ago

>> This is a simple self hosted server that has a simple but powerful interface to block ads, paywalls, and other nonsense. Specially for sites like medium, new york times which have paid articles that you normally cannot read. Now I do want you to support the creators you benefit from but if you just wanna see one single article and move on with your day then this might be helpful Personally I'm not a fan of this attitude. I've read and digested the arguments for it, but, for me, it runs close to "theft". For example, read the sentence again, but in the context of a restaurant. Sure I wanna support the creators, but what if I just want a single meal and then get on with my day? Businesses, including news web sites, need to monetize their content. There are a variety of ways they do that. You are free to consume their content or not. You either accept their monetization method as desirable or you do not. The "I just want to read one article" argument doesn't fly. If the article is so compelling, then follow their rules for accessing it. Yes, some sites behave badly. So stop visiting them. There is lots of free content on the web that is well presented and lacks corporate malfeasance. Read some of that instead. I get that I'm gonna get downvoted to oblivion with this post. HN readers are in love with ad blockers and paywall bypasses. But just because you can do something, just because you think it should be "free, no ads", does not make it right. Creators create. They get to choose how the world sees their creation. Support it, don't support it, that's up to you. Deciding to just take it anyway, on your terms (however righteous you feel you are) is not ok.

Daneel_ 2y ago
I agree with you. Where the gap lies for me is that I can’t just buy one meal, I have to sign up for the yearly meal plan even if I just want one meal. A few years ago I tracked how many times I visited some of the various paywalls sites for their articles, and typically it was between 5-10 times per year. One was 30, so I paid for a subscription to them, but I can’t justify several dozens of dollars for 5 articles on many other sites. If I can’t access their content because a bypass doesn’t work then so be it, however I wasn’t willing to pay for that content either. I feel like it’s the classic misconception regarding piracy by the movie industry - I wasn’t willing to pay money for it in the first place, so it’s not lost revenue (unfortunately). I was actually discussing this overall problem with my wife the other day, and I came to the conclusion that I basically want a “Netflix” but for news - I subscribe to one place and I get access to a whole range of publications. That’s worth it to me. I very much don’t see it happening though, sadly.

Animats 2y ago

The next step being 11ft 8 inches.[1] [1] http://11foot8.com/ http://11foot8.com/

willhackett 2y ago
Reminds me of Montague Street Bridge. [1] [1] https://howmanydayssincemontaguestreetbridgehasbeenhit.com/ https://howmanydayssincemontaguestreetbridgehasbeenhit.com/
Jenk 2y ago
> The next step Should that read "Previous"? :)
mvc 2y ago
14ft. Similar to 13ft but implemented in Rust.
- auadix 2y ago
  Or 4.2672m, for the rest of the world. :)
deleted 2y ago
[deleted]

drowntoge 2y ago

> Port 5000 is in use by another program. Either identify and stop that program, or start the server with a different port. An instruction on how to specify port would be nice.

RoseyWasTaken 2y ago
The docker-compose.yaml file is where you specify the ports you want to expose. It looks like by default it's 5000:5000 (5000 outside and inside the container). You will need to change it and then run docker-compose up -d. You can change it to something like 5133:5000 and access the instance through localhost:5133
- drowntoge 2y ago
  Thank you for the tip! I ended up editing the port parameter in the app.run() call within portable.py and it worked. Felt like it might be a good idea to add this as a runtime argument for easier customization.

Ikatza 2y ago

It's always seemed easier to me to use FF + Ublock + Bypass Paywalls. Never fails.

wasi_master 2y ago

Hello everyone, it's the author here. I initially created 13ft as a proof of concept, simply to test whether the idea would work. I never anticipated it would gain this much traction or become as popular as it has. I'm thrilled that so many of you have found it useful, and I'm truly grateful for all the support. Regarding the limitations of this approach, I'm fully aware that it isn't perfect, and it was never intended to be. It was just a quick experiment to see if the concept was feasible—and it seems that, at least sometimes, it is. Thank you all for the continued support.

darknavi 2y ago
Apologies for submitting it here if it caused any sense of being overwhelmed. Hopefully FOSS is supportive here instead of overwhelming. Thanks for sharing the project with the internet!

ck2 2y ago

Just a caution you are probably headed for this result if hosted on GitHub (or GitLab) https://github.com/bpc-clone/bypass-paywalls-clean-filters https://github.com/bpc-clone/bypass-paywalls-clean-filters https://news.ycombinator.com/item?id=40007670 https://news.ycombinator.com/item?id=40007670 https://news.ycombinator.com/item?id=40015961 https://news.ycombinator.com/item?id=40015961

efangs 2y ago

From my experience, pihole is very easy to setup for this use case: https://pi-hole.net/ https://pi-hole.net/

deleted 2y ago
[deleted]

martin82 2y ago

I'm surprised this works at all. What sysadmin is so naive to rely on the Googlebot useragent?? Doesn't everyone fetch the official list of Googlebot IPs and then add those to a whitelist?