3 ms·

Most paywalls just allow search engines to read their content just fine. Because they do want discoverability, they want their cake and eat it. There's a few p

Most paywalls just allow search engines to read their content just fine. Because they do want discoverability, they want their cake and eat it. There's a few publications that don't even do that though and archive.is is very good at bypassing them so I do imagine they use logins for those, but for the masses of sites it's not currently necessary.

direwolf20 9mo ago
You can't impersonate Google. Sites check the source IP and they don't overlap with Google Cloud.
- chrisjj 9mo ago
  You can for sites that can't afford the cost of keeping up-to-date with the Google IP list without which they can lose timely indexing. That is many.
  - otterley 9mo ago
    What do you mean by “afford the cost”? The list is free of charge (https://support.google.com/a/answer/10026322?hl=en-GB https://support.google.com/a/answer/10026322?hl=en-GB) and maintenance can be fully automated.
    - chrisjj 9mo ago
      I mean cost of server setup and execution.
      - otterley 9mo ago
        The server that is providing the content exists already. That's a sunk cost.
        chrisjj 9mo ago
        "setup and execution".
        otterley 9mo ago
        What serious operator of a service isn't budgeting time to implement and operate critical maintenance functions?
        chrisjj 9mo ago
        Me for one. Adding an auto-updating IP address blocker to my personal blog site would probably cost more than setting up the whole site did in the first place.
        mr_mitm 9mo ago
        We're talking about sites that make their living via subscriptions. They should have a great interest at blocking archive.is, which is, by the way, the only service that can reliably bypass many paywalls. Clearly whatever they're doing is not easily replicated.
        chrisjj 9mo ago
        > We're talking about sites that make their living via subscriptions. Sorry, but I wasn't. I thought that was clear from "can't afford the cost of keeping up-to-date with the Google IP list". > They should have a great interest at blocking archive.is Agreed, and many should have a budget to suit. So I conclude archive.is has put a lot of effort and cost into its defence. And all for free to us, the users.
        otterley 9mo ago
        Have you actually priced it, or are you just guessing? Are you doing regular patching? Automated restarts? Watching for security breaches? Or just praying it stays up forever? Otherwise, respectfully, I would not classify you as a "serious operator." Your site could live or die, and it would be all the same to you. Or, you've handed it to a third party for management and they don't offer much in the way of resilience or stability.
- wolvoleo 9mo ago
  Google isn't the only search engine in the world of course. It probably is pretty much the only one that matters in America but the world is not just America either.
  - direwolf20 9mo ago
    It's the only one websites don't block. That's one reason it's so hard to make another search engine.
mr_mitm 9mo ago
Then why hasn't anyone built a client-side browser addon that impersonates a suitable search engine?
- wolvoleo 9mo ago
  They have. It's called bypass-paywalls-clean . It works pretty ok. It just keeps getting banned from the addon catalogs because of complaints from media. The Firefox one was taken down by a french newspaper. So you have to sideload it, which is hard to do on Android. Edit: it looks like even the github was taken down now: https://github.com/iamadamdev/bypass-paywalls-firefox https://github.com/iamadamdev/bypass-paywalls-firefox But yes it exists. And it works for most sites. It's just hard to get it now.
  - eipi10_hn 9mo ago
    It's on gitflic.ru now.
    - wolvoleo 9mo ago
      Hmm yeah but their adversaries did achieve their goal by pushing it away from the mainstream sites. Now we're into this situation of "how much do I trust this vague Russian site with my browsing activity". At least the addon declares the sites it's for and ignores the rest but still I'm a lot less comfortable with it. It's more something I'd install in a container now, limiting its usefulness :( In practice I just use archive.today now.
      - eipi10_hn 9mo ago
        Yeah it's unavoidable. Bypassing paywalls is not a good idea for tools that depend on browsers' stores distribution.

direwolf20 9mo ago

You can't impersonate Google. Sites check the source IP and they don't overlap with Google Cloud.

chrisjj 9mo ago
You can for sites that can't afford the cost of keeping up-to-date with the Google IP list without which they can lose timely indexing. That is many.
- otterley 9mo ago
  What do you mean by “afford the cost”? The list is free of charge (https://support.google.com/a/answer/10026322?hl=en-GB https://support.google.com/a/answer/10026322?hl=en-GB) and maintenance can be fully automated.
  - chrisjj 9mo ago
    I mean cost of server setup and execution.
    - otterley 9mo ago
      The server that is providing the content exists already. That's a sunk cost.
      - chrisjj 9mo ago
        "setup and execution".
        otterley 9mo ago
        What serious operator of a service isn't budgeting time to implement and operate critical maintenance functions?
        chrisjj 9mo ago
        Me for one. Adding an auto-updating IP address blocker to my personal blog site would probably cost more than setting up the whole site did in the first place.
        mr_mitm 9mo ago
        We're talking about sites that make their living via subscriptions. They should have a great interest at blocking archive.is, which is, by the way, the only service that can reliably bypass many paywalls. Clearly whatever they're doing is not easily replicated.
        chrisjj 9mo ago
        > We're talking about sites that make their living via subscriptions. Sorry, but I wasn't. I thought that was clear from "can't afford the cost of keeping up-to-date with the Google IP list". > They should have a great interest at blocking archive.is Agreed, and many should have a budget to suit. So I conclude archive.is has put a lot of effort and cost into its defence. And all for free to us, the users.
        otterley 9mo ago
        Have you actually priced it, or are you just guessing? Are you doing regular patching? Automated restarts? Watching for security breaches? Or just praying it stays up forever? Otherwise, respectfully, I would not classify you as a "serious operator." Your site could live or die, and it would be all the same to you. Or, you've handed it to a third party for management and they don't offer much in the way of resilience or stability.
wolvoleo 9mo ago
Google isn't the only search engine in the world of course. It probably is pretty much the only one that matters in America but the world is not just America either.
- direwolf20 9mo ago
  It's the only one websites don't block. That's one reason it's so hard to make another search engine.

mr_mitm 9mo ago

Then why hasn't anyone built a client-side browser addon that impersonates a suitable search engine?

wolvoleo 9mo ago
They have. It's called bypass-paywalls-clean . It works pretty ok. It just keeps getting banned from the addon catalogs because of complaints from media. The Firefox one was taken down by a french newspaper. So you have to sideload it, which is hard to do on Android. Edit: it looks like even the github was taken down now: https://github.com/iamadamdev/bypass-paywalls-firefox https://github.com/iamadamdev/bypass-paywalls-firefox But yes it exists. And it works for most sites. It's just hard to get it now.
- eipi10_hn 9mo ago
  It's on gitflic.ru now.
  - wolvoleo 9mo ago
    Hmm yeah but their adversaries did achieve their goal by pushing it away from the mainstream sites. Now we're into this situation of "how much do I trust this vague Russian site with my browsing activity". At least the addon declares the sites it's for and ignores the rest but still I'm a lot less comfortable with it. It's more something I'd install in a container now, limiting its usefulness :( In practice I just use archive.today now.
    - eipi10_hn 9mo ago
      Yeah it's unavoidable. Bypassing paywalls is not a good idea for tools that depend on browsers' stores distribution.