4 ms·

Going a step further, I think google should stop crawling websites that are paywalled. When I search for something, I want to see results that I can click on. N

Going a step further, I think google should stop crawling websites that are paywalled. When I search for something, I want to see results that I can click on. Not some snippets from NYT, WSJ, Bloomberg and others which are heavily paywalled.

carlosjobim 3y ago
When I search for something I want the best results for the query, and sometimes those are behind a paywall.
- JoshTriplett 3y ago
  Checkbox, unchecked by default: [ ] I want to see results from sites I'd have to pay to access
  - kccqzy 3y ago
    The obvious problem to this armchair expert "solution" is that Google doesn't know what I am already paying. I pay NYT for a subscription, but Google doesn't know that. For obvious privacy reasons users don't want to tell Google what sites they already have subscriptions with. And I don't even log in to Google to do a search so there's no place to store that information even if I actively wanted to provide that to Google.
    - JoshTriplett 3y ago
      I wasn't presenting it as a perfect solution. I was observing that many people don't want to see paywalled results, some may want to see all of them because they may choose to pay, some may want to see them because they plan to use a paywall bypass, and some as you pointed out may want to see the subset they already pay for but not others. As a first pass, a binary approach seems better than nothing, and is simpler to provide than a more complex user-subscription-specific solution.
- pipo234 3y ago
  I'd pay a little if we could get rid of the pop ups and cookie banners, advertisements and click bait content. But after 30 years we're still missing the infrastructure for micro payments. For news sites and netflix we now have subscriptions shielded by paywals, which really is incompatible with hyperlinked sites or search engines. Even if you subscribed to 1000 services, the experience would probably be horrible. The internet was designed to be free, but evidently that's not a good business model if you want to make a living.
  - carlosjobim 3y ago
    > But after 30 years we're still missing the infrastructure for micro payments. Sad but true. The closest we have to a solution right now is PressReader, which is just too expensive in my opinion at $30 per month.
    - pipo234 3y ago
      Micro payments only make sense if the intrinsic transaction costs are (near) zero. Like reaching for your wallet to pick a coin to give to a homeless person. With trusted third parties or block chains, the transaction costs are unfortunately much, much higher, especially initially without proper scales economics.
      - carlosjobim 3y ago
        It could be as simple for the user as a button in your web browser to donate or pay for the site currently open in your tab. And a third party intermediary that once a month collects payment/donations and distributes. The problem with Visa/MC is the minimum transaction fee of 10-25c But a lot of work and planning would be needed to get anywhere with such an idea.
- admax88qqq 3y ago
  If I'm not willing to pay for a paywall then the "best" content is not behind a paywall, because I won't read it. My definition of "best" includes my ability to actually read the content under my terms.
  - carlosjobim 3y ago
    If I ask you what is the best restaurant in town, will you answer that it's your mommas house because there you always eat for free? Cost has nothing to do with determining the quality of a search result, and search engines shouldn't discriminate against paywalled content. But I think it's a good idea to let users like you check a box to hide paywalled results.
    - admax88qqq 3y ago
      That's an insane take on what I said. No, I won't answer my mommas house because you can't just show up and eat there as a restauraunt. It doesn't fit the search criteria you asked for. If you ask what the best restauraunt in town is, I'll give you a restauraunt. If you ask what the best restauruant in town is, I won't send you to a place where you can buy a "best restauraunts in town guide" for 5 bucks, because that's not what you're asking for.
    - chongli 3y ago
      There are millions of websites out there that are free for anyone to read, including this one! Restaurants that serve free meals are not the norm, so this analogy doesn't make a lot of sense. If every website was paywalled and required a subscription, like cable TV channels, then you might have a point.
      - carlosjobim 3y ago
        People use search engines professionally and not only for entertainment. There is an icebergs worth of important and valuable information online behind paywalls, not only articles or news. Information workers use a search engine to find the information they need, pay the cost if it's paywalled, and then cancel any subscription after getting what they needed. Long gone are the days of "surfing the web", when most of us spent our time online just randomly browsing around.
- nonameiguess 3y ago
  At a certain point, if you want a subscription service, why wouldn't you just do something like suscribe to Bloomberg News, then get all your news by going directly to their site rather than going through a search engine or aggregator. If you're looking at an aggregator, inherently you want to see many possible sources, including ones you may only read once a year. Nobody is going to subscribe to hundreds of separate sources individually just to read them once in a blue moon. Ironically, the predatory and terrible academic journal industry is probably the only thing out there right now that comes close to getting this right. Rather than expecting anyone to subscribe to each journal individually, they give a bulk subscription to an entire publishing service that then grants access to many journals. If someone out there offered a $20 a month service that granted access to Bloomberg, NY Times, Wall Street Journal, Economist, WaPost, Financial Times, all in one, I'd gladly buy that. But there is no way in hell I'm subscribing to all of those separately. Even if the aggregate price was cheaper, I wouldn't want to do that.
  - carlosjobim 3y ago
    1. There is much paywalled information online that is not in the form of a subscription, but instead one-time fees. 2. Just because you paid at the paywall, doesn't mean you have to subscribe for life. You can pay to get the information you need and then instantly cancel any subscription. > If someone out there offered a $20 a month service that granted access to Bloomberg, NY Times, Wall Street Journal, Economist, WaPost, Financial Times, all in one, I'd gladly buy that. PressReader is pretty much this, although the price is $30 and not $20.
matbilodeau 3y ago
At least the "archive" workaround still works
- pessimizer 3y ago
  It's piracy, though. I'm sure plenty of paywalled outlets would be happy to see it gone.
  - jupp0r 3y ago
    It's not piracy, they explicitly allow this.
kccqzy 3y ago
The open web has deteriorated to such an extent that either information is paywalled or free but has commercial motivations behind it (affiliate links, sponsorships). It turns out there's little free, non-commercial, high quality content on the web.
- shadowgovt 3y ago
  > turns out there's little free, non-commercial, high quality content on the web. Well, sure. Who pays for it?
  - kccqzy 3y ago
    No one. That's why I commented to make the point that Google should not refuse to crawl commercial content.
- pipo234 3y ago
  True. There is some great content (blogs, OS software, books, p2p networks, public libraries, academia, government), but by and large that wasn't created with monetary incentives. Could there be a way to sustainably make enough money from visitors without making it all suck?
  - kccqzy 3y ago
    I think there really needs to be a micro-transaction mechanism on the web, but unfortunately it was needed ten years ago and there still isn't one.
    - golem14 3y ago
      Yes but the crypto folks didn't do that, they were busy chasing nfts.
      - chroma 3y ago
        There have been a lot of ridiculous things in cryptocurrency land, but what's wrong with Basic Attention Token? It has been around for six years and seems to solve the problem of paying content creators via microtransactions. The only wrinkle is that finance laws force everyone to verify their identity with a government-issued ID before transacting BAT.[1] This is a problem for all microtransactions, not just cryptocurrency. 1. https://support.brave.com/hc/en-us/articles/360032158891-What-is-KYC-and-how-does-it-fit-into-Brave-Rewards- https://support.brave.com/hc/en-us/articles/360032158891-Wha...
renewiltord 3y ago
You can easily write this as a Chrome extension if you so desire.
chroma 3y ago
Google isn't circumventing paywalls. Most paywalled sites whitelist search engines so that their content gets crawled and more people visit them.

carlosjobim 3y ago

When I search for something I want the best results for the query, and sometimes those are behind a paywall.

JoshTriplett 3y ago
Checkbox, unchecked by default: [ ] I want to see results from sites I'd have to pay to access
- kccqzy 3y ago
  The obvious problem to this armchair expert "solution" is that Google doesn't know what I am already paying. I pay NYT for a subscription, but Google doesn't know that. For obvious privacy reasons users don't want to tell Google what sites they already have subscriptions with. And I don't even log in to Google to do a search so there's no place to store that information even if I actively wanted to provide that to Google.
  - JoshTriplett 3y ago
    I wasn't presenting it as a perfect solution. I was observing that many people don't want to see paywalled results, some may want to see all of them because they may choose to pay, some may want to see them because they plan to use a paywall bypass, and some as you pointed out may want to see the subset they already pay for but not others. As a first pass, a binary approach seems better than nothing, and is simpler to provide than a more complex user-subscription-specific solution.
pipo234 3y ago
I'd pay a little if we could get rid of the pop ups and cookie banners, advertisements and click bait content. But after 30 years we're still missing the infrastructure for micro payments. For news sites and netflix we now have subscriptions shielded by paywals, which really is incompatible with hyperlinked sites or search engines. Even if you subscribed to 1000 services, the experience would probably be horrible. The internet was designed to be free, but evidently that's not a good business model if you want to make a living.
- carlosjobim 3y ago
  > But after 30 years we're still missing the infrastructure for micro payments. Sad but true. The closest we have to a solution right now is PressReader, which is just too expensive in my opinion at $30 per month.
  - pipo234 3y ago
    Micro payments only make sense if the intrinsic transaction costs are (near) zero. Like reaching for your wallet to pick a coin to give to a homeless person. With trusted third parties or block chains, the transaction costs are unfortunately much, much higher, especially initially without proper scales economics.
    - carlosjobim 3y ago
      It could be as simple for the user as a button in your web browser to donate or pay for the site currently open in your tab. And a third party intermediary that once a month collects payment/donations and distributes. The problem with Visa/MC is the minimum transaction fee of 10-25c But a lot of work and planning would be needed to get anywhere with such an idea.
admax88qqq 3y ago
If I'm not willing to pay for a paywall then the "best" content is not behind a paywall, because I won't read it. My definition of "best" includes my ability to actually read the content under my terms.
- carlosjobim 3y ago
  If I ask you what is the best restaurant in town, will you answer that it's your mommas house because there you always eat for free? Cost has nothing to do with determining the quality of a search result, and search engines shouldn't discriminate against paywalled content. But I think it's a good idea to let users like you check a box to hide paywalled results.
  - admax88qqq 3y ago
    That's an insane take on what I said. No, I won't answer my mommas house because you can't just show up and eat there as a restauraunt. It doesn't fit the search criteria you asked for. If you ask what the best restauraunt in town is, I'll give you a restauraunt. If you ask what the best restauruant in town is, I won't send you to a place where you can buy a "best restauraunts in town guide" for 5 bucks, because that's not what you're asking for.
  - chongli 3y ago
    There are millions of websites out there that are free for anyone to read, including this one! Restaurants that serve free meals are not the norm, so this analogy doesn't make a lot of sense. If every website was paywalled and required a subscription, like cable TV channels, then you might have a point.
    - carlosjobim 3y ago
      People use search engines professionally and not only for entertainment. There is an icebergs worth of important and valuable information online behind paywalls, not only articles or news. Information workers use a search engine to find the information they need, pay the cost if it's paywalled, and then cancel any subscription after getting what they needed. Long gone are the days of "surfing the web", when most of us spent our time online just randomly browsing around.
nonameiguess 3y ago
At a certain point, if you want a subscription service, why wouldn't you just do something like suscribe to Bloomberg News, then get all your news by going directly to their site rather than going through a search engine or aggregator. If you're looking at an aggregator, inherently you want to see many possible sources, including ones you may only read once a year. Nobody is going to subscribe to hundreds of separate sources individually just to read them once in a blue moon. Ironically, the predatory and terrible academic journal industry is probably the only thing out there right now that comes close to getting this right. Rather than expecting anyone to subscribe to each journal individually, they give a bulk subscription to an entire publishing service that then grants access to many journals. If someone out there offered a $20 a month service that granted access to Bloomberg, NY Times, Wall Street Journal, Economist, WaPost, Financial Times, all in one, I'd gladly buy that. But there is no way in hell I'm subscribing to all of those separately. Even if the aggregate price was cheaper, I wouldn't want to do that.
- carlosjobim 3y ago
  1. There is much paywalled information online that is not in the form of a subscription, but instead one-time fees. 2. Just because you paid at the paywall, doesn't mean you have to subscribe for life. You can pay to get the information you need and then instantly cancel any subscription. > If someone out there offered a $20 a month service that granted access to Bloomberg, NY Times, Wall Street Journal, Economist, WaPost, Financial Times, all in one, I'd gladly buy that. PressReader is pretty much this, although the price is $30 and not $20.

matbilodeau 3y ago

At least the "archive" workaround still works

pessimizer 3y ago
It's piracy, though. I'm sure plenty of paywalled outlets would be happy to see it gone.
- jupp0r 3y ago
  It's not piracy, they explicitly allow this.

kccqzy 3y ago

The open web has deteriorated to such an extent that either information is paywalled or free but has commercial motivations behind it (affiliate links, sponsorships). It turns out there's little free, non-commercial, high quality content on the web.

shadowgovt 3y ago
> turns out there's little free, non-commercial, high quality content on the web. Well, sure. Who pays for it?
- kccqzy 3y ago
  No one. That's why I commented to make the point that Google should not refuse to crawl commercial content.
pipo234 3y ago
True. There is some great content (blogs, OS software, books, p2p networks, public libraries, academia, government), but by and large that wasn't created with monetary incentives. Could there be a way to sustainably make enough money from visitors without making it all suck?
- kccqzy 3y ago
  I think there really needs to be a micro-transaction mechanism on the web, but unfortunately it was needed ten years ago and there still isn't one.
  - golem14 3y ago
    Yes but the crypto folks didn't do that, they were busy chasing nfts.
    - chroma 3y ago
      There have been a lot of ridiculous things in cryptocurrency land, but what's wrong with Basic Attention Token? It has been around for six years and seems to solve the problem of paying content creators via microtransactions. The only wrinkle is that finance laws force everyone to verify their identity with a government-issued ID before transacting BAT.[1] This is a problem for all microtransactions, not just cryptocurrency. 1. https://support.brave.com/hc/en-us/articles/360032158891-What-is-KYC-and-how-does-it-fit-into-Brave-Rewards- https://support.brave.com/hc/en-us/articles/360032158891-Wha...

renewiltord 3y ago

You can easily write this as a Chrome extension if you so desire.

chroma 3y ago

Google isn't circumventing paywalls. Most paywalled sites whitelist search engines so that their content gets crawled and more people visit them.