8 ms·

We're moving progressively in the direction of "pages can't be served for free anymore". Which, I don't think is a problem, and in fact I think it's something w

by Taek 1y ago

We're moving progressively in the direction of "pages can't be served for free anymore". Which, I don't think is a problem, and in fact I think it's something we should have addressed a long time ago. Cloudflare only needs to exist because the server doesn't get paid when a user or bot requests resources. Advertising only needs to exist because the publisher doesn't get paid when a user or bot requests resources. And the thing is... people already pay for internet. They pay their ISP. So people are perfectly happy to pay for resources that they consume on the Internet, and they already have an infrastructure for doing so. I feel like the answer is that all web requests should come with a price tag, and the ISP that is delivering the data is responsible for paying that price tag and then charging the downstream user. It's also easy to ratelimit. The ISP will just count the price tag as 'bytes'. So your price could be 100 MB or whatever (independent of how large the response is), and if your internet is 100 mbps, the ISP will stall out the request for 8 seconds, and then make it. If the user aborts the request before the page loads, the ISP won't send the request to the server and no resources are consumed.

Forgeties79 1y ago
My first reaction: This solution would basically kill what little remaining fun there is to be had browsing the Internet and all but assure no new sites/smaller players will ever see traffic. Curious to hear other perspectives here. Maybe I’m over reacting/misunderstanding.
- armchairhacker 1y ago
  Depending on the implementation (a big if) it would help smaller websites, because it would make hosting much cheaper. ISPs don’t choose what sites users visit, only what they pay. As long as the ISP isn’t giving significant discounts to visiting big sites (just charging a fixed rate per bytes downloads and uploaded) and charging something reasonable, visiting a small site would be so cheap (a few cents at most, but more likely <1 cent) users won’t weigh cost at all.
  - Forgeties79 1y ago
    But users depend on major sites like google [insert service] still and will prioritize their usage accordingly like limited minutes and texts back in the day, right?
    - armchairhacker 1y ago
      Networking is so cheap, unless ISPs drastically inflate their price, users won’t care. The average American allegedly* downloads 650-700GB/month, or >20GB/day. 10MB is more than enough for a webpage (honestly, 1MB is usually enough), so that means on average, ISPs serve over 2000 webpages worth of data per day. And the average internet plan is allegedly** $73/month, or <$2.50/day. So $2.50 gets you over 2000 indie sites. That’s cheap enough, wrapped in a monthly bill, users won’t even pay attention to what sites they visit. The only people hurt by an ideal (granted, ideal) implementation are those who abuse fixed rates and download unreasonable amounts of data, like web crawlers who visit the same page seconds apart for many pages in parallel. * https://www.astound.com/learn/internet/average-internet-data-usage/#statistics https://www.astound.com/learn/internet/average-internet-data... ** https://www.nerdwallet.com/article/finance/how-much-is-internet-per-month https://www.nerdwallet.com/article/finance/how-much-is-inter...
      - inquirerGeneral 1y ago
        [dead]
      - brookst 1y ago
        Wait, so the ISPs do from taking $73/user home today to taking $0/user home tomorrow under this plan?
        Forgeties79 1y ago
        Yeah same reaction here - there's no world in which ISP's would agree to this and even if they did I don't want to add them to my list of utilities I have to regularly fight with over claimed vs. actual usage like I do with my power/water/gas companies.
- Analemma_ 1y ago
  If site operators can’t afford the costs of keeping sites up in the face of AI scraping, the new/smaller sites are gone anyway.
  - Forgeties79 1y ago
    Maybe not but we are not realistically in an either/or scenario here.
dabockster 1y ago
> We're moving progressively in the direction of "pages can't be served for free anymore". Which, I don't think is a problem, and in fact I think it's something we should have addressed a long time ago. I agree, but your idea below that is overly complicated. You can't micro-transact the whole internet. That idea feels like those episodes of Star Trek DS9 that take place on Feregenar - where you have to pay admission and sign liability wavers to even walk on the sidewalk outside. It's not a true solution.
- vineyardmike 1y ago
  > You can't micro-transact the whole internet. I agree that end-users cannot handle micro transactions across the whole internet. That said, I would like to point out that most of the internet is blanketed in ads and ads involve tons of tiny quick auctions and micro transactions that occur on each page load. It is totally possible for a system to evolve involving tons of tiny transactions across page loads.
  - edoceo 1y ago
    Remember Flattr?
  - helloplanets 1y ago
    You could argue that the suggested system is actually much simpler than the one we currently have for the sites that are "free", aka funded with ads. The lengths Meta and the like go to in order to maximize clickthroughs...
- sellmesoap 1y ago
  > You can't micro-transact the whole internet. Clearly you don't have the lobes for business /s
- Taek 1y ago
  The presented solution has invisible UX via layering it into existing metered billing. And, the whole internet is already micro-transactioned! Every page with ads is doing a bidding war and spending money on your attention. The only person not allowed to bid is yourself!
seer 1y ago
Hah still remember the old “solving the internet with hate” idea from Zed Shaw in the glory days of Ruby on Rails. https://weblog.masukomi.org/2018/03/25/zed-shaws-utu-saving-the-internet-with-hate/ https://weblog.masukomi.org/2018/03/25/zed-shaws-utu-saving-... I do believe we will end there eventually, with the emerging tech like Brazil’s and India’s payment architectures it should be a possibility in the coming decades
debesyla 1y ago
Wouldn't this lead to pirated page clones where customer pays less for same-ish content, and less, all the way down to essentially free? Because I as an user would be glad to have "free sites only" filter, and then just steal content :)) But it's an interesting idea and thought experiment.
- armchairhacker 1y ago
  That’s fine. The point for website owners isn’t to make money, it’s to not spend money hosting (or more specifically, to pay a small fixed rate hosting). They want people to see the content; if someone makes the content more accessible, that’s a good thing.
  - mapontosevenths 1y ago
    You ignore the issue of motivation. Most web content exists because someone wants to make money on it. If the content creator can't do that, they will stop producing content. These AI web crawlers (Google, Perplexity, etc) are self-cannibalizing robots. They eat the goose that laid the golden egg for breakfast, and lose money doing it most of the time. If something isn't done to incentivize content creators again eventually there will be only walled-gardens and obsolete content left for the cannibals.
    - armchairhacker 1y ago
      AFAIK, currently creators get money while not charging for users because of ads. While I don’t blame creators for using ads now, I don’t think they’re a long-term solution. Ads are already blocked when people visit the site with ad blockers, which are becoming more popular. Obvious sponsored content may be blocked with the ads, and non-obvious sponsored content turns these “creators” into “shills” who are inauthentic and untrustworthy. Even without Google summaries, ad revenue may decrease over time as advertisers realize they aren’t effective or want more profit; even if it doesn’t, it’s my personal opinion that society should decrease the overall amount of ads. Not everyone creates only for money, the best only create for enough money to sustain themselves. A long-term solution is to expand art funding (e.g. creators apply for grants with their ideas and, if accepted, get paid a fixed rate to execute them) or UBI. Then media can be redistributed, remixed, etc. without impacting creators’ finances.
nazcan 1y ago
I think value is not proportional to bytes - an AI only needs to read a page once to add it to its model, and then served the effectively cached data many times.
Terretta 1y ago
Or, flip this, don't expect to get paid for pamphleteering?
novok 1y ago
The reason why that didn’t work was because regulations made micropayments too expensive, and the government wants it that way to keep control over the financial system.
OptionOfT 1y ago
> We're moving progressively in the direction of "pages can't be served for free anymore". Which, I don't think is a problem, and in fact I think it's something we should have addressed a long time ago. But it's done through a bait and switch. They serve the full article to Google, which allows Google to show you excerpts that you have to pay for. It would be better if Google shows something like PAYMENT REQUIRED on top, at least that way I know what I'm getting at.
- mh- 1y ago
  > They serve the full article to Google, which allows Google to show you excerpts that you have to pay for. I'm old enough to remember when that was grounds for getting your site removed from Google results - "cloaking" was against the rules. You couldn't return one result for Googlebot, and another for humans. No idea when they stopped doing that, but they obviously have let go of that principle.
  - dspillett 1y ago
    I remember that too, along with high-profile punishments for sites that were keyword stuffing (IIRC a couple of decades ago BMW were completely unlisted for a time for this reason). I think it died largely because it became impossible top police with any reliability, and being strict about it would remove too much from Google's index because many sites are not easily indexable without them providing a “this is the version without all the extra round-trips for ad impressions and maybe a login needed” variant to common search engines. Applying the rule strictly would mean that sites implementing PoW tricks like Anubis to reduce unwanted bot traffic would not be included in the index if they serve to Google without the PoW step. I can't say I like that this has been legitimised even for the (arguably more common) deliberate bait & switch tricks is something I don't like, but (I think) I understand why the rule was allowed to slide.
chromatin 1y ago
402 Payment Required https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/402 https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/... Sadly development along these lines has not progressed. Yes, Google Cloud and other services may return it and require some manual human intervention, but I'd love to see _automatic payment negotiation_. I'm hopeful that instant-settlement options like Bitcoin Lightning payments could progress us past this. https://docs.lightning.engineering/the-lightning-network/l402 https://docs.lightning.engineering/the-lightning-network/l40... https://hackernoon.com/the-resurgence-of-http-402-in-the-age-of-agentic-ai https://hackernoon.com/the-resurgence-of-http-402-in-the-age...
makingstuffs 1y ago
As time passes I’m more certain in the belief that the internet will end up being a licensed system with insanely high barriers to entry which will stop your average dev from even being able to afford deploying a hobby project on it. Your idea of micro transacting web requests would play into it and probably end up with a system like Netflix where your ISP has access to a set of content creators to whom they grant ‘unlimited’ access as part of the service fee. I’d imagine that accessing any content creators which are not part of their package will either be blocked via a paywall (buy an addon to access X creators outside our network each month) or charged at an insane price per MB as is the case with mobile data. Obvious this is all super hypothetical but weirder stuff has happened in my lifetime
AlexandrB 1y ago
A scary observation in light of another front page article right now: https://news.ycombinator.com/item?id=44783566 https://news.ycombinator.com/item?id=44783566 If pages can't be served for free, all internet content is at the mercy of payment processors and their ideas of "brand safety".
- inquirerGeneral 1y ago
  [dead]
- dspillett 1y ago
  “Free” could have a number of meanings here. Free to the viewer, free to the hoster, free to the creator, etc… That content can't be served entirely for free doesn't mean that all content will require payment, and so is subject to issues with payment processors, just that some things may gravitate back to a model where it costs a small amount to host something (i.e. pay for home internet and host bits off that, or you might have VPS out there that runs tools and costs a few $ /yr or /month). I pay for resources to host my bits & bobs instead of relying on services provided in exchange for stalking the people looking at them, this is free for the viewer as they aren't even paying indirectly. Most things are paid for anyway, even if the person hosting it nor my looking at it are paying directly: adtech arseholes give services to people hosting content in exchange for the ability to stalk us and attempt to divert our attention. Very few sites/apps, other than play/hobby ones like mine or those from more actively privacy focused types, are free of that.
- Taek 1y ago
  That's already a deep problem for all of society. If we don't want that to be an ongoing issue, we need to make sure money is a neutral infrastructure. It doesn't just apply to the web, it applies to literally everything that we spend money on via a third party service. Which is... most everything these days.
bboygravity 1y ago
I get your thinking, but x.com is proof that simply making users pay (quite a lot) does not eliminate bots. The amount of "verified" paying "users" with a blue checkmark that are just total LLM bots is incredible on there. As long as spamming and DDOS'ing pays more than whatever the request costs, it will keep existing.
Saline9515 1y ago
Why would I pay for a page if I don't know if the content is what I asked for? How much are you going to pay? How much are you going to charge? This will end up in SEO hell, especially with AI-generated pages farming paid clicks.
adrian_b 1y ago
Your theory does not match the practice of Cloudflare. Whatever method is used by Cloudflare for detecting "threats" has nothing to do with consuming resources on the "protected" servers. The so-called "threats" are identified in users that may make a few accesses per day to a site, transferring perhaps a few kilobytes of useful data on the viewed pages (besides whatever amount of stupid scripts the site designer has implemented). So certainly Cloudflare does not meter the consumed resources. Moreover, Cloudflare preemptively annoys any user who accesses for the first time a site, having never consumed any resources, perhaps based on irrational profiling based on the used browser and operating system, and geographical location.

by Taek 1y ago

Forgeties79 1y ago

My first reaction: This solution would basically kill what little remaining fun there is to be had browsing the Internet and all but assure no new sites/smaller players will ever see traffic. Curious to hear other perspectives here. Maybe I’m over reacting/misunderstanding.

armchairhacker 1y ago
Depending on the implementation (a big if) it would help smaller websites, because it would make hosting much cheaper. ISPs don’t choose what sites users visit, only what they pay. As long as the ISP isn’t giving significant discounts to visiting big sites (just charging a fixed rate per bytes downloads and uploaded) and charging something reasonable, visiting a small site would be so cheap (a few cents at most, but more likely <1 cent) users won’t weigh cost at all.
- Forgeties79 1y ago
  But users depend on major sites like google [insert service] still and will prioritize their usage accordingly like limited minutes and texts back in the day, right?
  - armchairhacker 1y ago
    Networking is so cheap, unless ISPs drastically inflate their price, users won’t care. The average American allegedly* downloads 650-700GB/month, or >20GB/day. 10MB is more than enough for a webpage (honestly, 1MB is usually enough), so that means on average, ISPs serve over 2000 webpages worth of data per day. And the average internet plan is allegedly** $73/month, or <$2.50/day. So $2.50 gets you over 2000 indie sites. That’s cheap enough, wrapped in a monthly bill, users won’t even pay attention to what sites they visit. The only people hurt by an ideal (granted, ideal) implementation are those who abuse fixed rates and download unreasonable amounts of data, like web crawlers who visit the same page seconds apart for many pages in parallel. * https://www.astound.com/learn/internet/average-internet-data-usage/#statistics https://www.astound.com/learn/internet/average-internet-data... ** https://www.nerdwallet.com/article/finance/how-much-is-internet-per-month https://www.nerdwallet.com/article/finance/how-much-is-inter...
    - inquirerGeneral 1y ago
      [dead]
    - brookst 1y ago
      Wait, so the ISPs do from taking $73/user home today to taking $0/user home tomorrow under this plan?
      - Forgeties79 1y ago
        Yeah same reaction here - there's no world in which ISP's would agree to this and even if they did I don't want to add them to my list of utilities I have to regularly fight with over claimed vs. actual usage like I do with my power/water/gas companies.
Analemma_ 1y ago
If site operators can’t afford the costs of keeping sites up in the face of AI scraping, the new/smaller sites are gone anyway.
- Forgeties79 1y ago
  Maybe not but we are not realistically in an either/or scenario here.

dabockster 1y ago

> We're moving progressively in the direction of "pages can't be served for free anymore". Which, I don't think is a problem, and in fact I think it's something we should have addressed a long time ago. I agree, but your idea below that is overly complicated. You can't micro-transact the whole internet. That idea feels like those episodes of Star Trek DS9 that take place on Feregenar - where you have to pay admission and sign liability wavers to even walk on the sidewalk outside. It's not a true solution.

vineyardmike 1y ago
> You can't micro-transact the whole internet. I agree that end-users cannot handle micro transactions across the whole internet. That said, I would like to point out that most of the internet is blanketed in ads and ads involve tons of tiny quick auctions and micro transactions that occur on each page load. It is totally possible for a system to evolve involving tons of tiny transactions across page loads.
- edoceo 1y ago
  Remember Flattr?
- helloplanets 1y ago
  You could argue that the suggested system is actually much simpler than the one we currently have for the sites that are "free", aka funded with ads. The lengths Meta and the like go to in order to maximize clickthroughs...
sellmesoap 1y ago
> You can't micro-transact the whole internet. Clearly you don't have the lobes for business /s
Taek 1y ago
The presented solution has invisible UX via layering it into existing metered billing. And, the whole internet is already micro-transactioned! Every page with ads is doing a bidding war and spending money on your attention. The only person not allowed to bid is yourself!

seer 1y ago

Hah still remember the old “solving the internet with hate” idea from Zed Shaw in the glory days of Ruby on Rails. https://weblog.masukomi.org/2018/03/25/zed-shaws-utu-saving-the-internet-with-hate/ https://weblog.masukomi.org/2018/03/25/zed-shaws-utu-saving-... I do believe we will end there eventually, with the emerging tech like Brazil’s and India’s payment architectures it should be a possibility in the coming decades

debesyla 1y ago

Wouldn't this lead to pirated page clones where customer pays less for same-ish content, and less, all the way down to essentially free? Because I as an user would be glad to have "free sites only" filter, and then just steal content :)) But it's an interesting idea and thought experiment.

armchairhacker 1y ago
That’s fine. The point for website owners isn’t to make money, it’s to not spend money hosting (or more specifically, to pay a small fixed rate hosting). They want people to see the content; if someone makes the content more accessible, that’s a good thing.
- mapontosevenths 1y ago
  You ignore the issue of motivation. Most web content exists because someone wants to make money on it. If the content creator can't do that, they will stop producing content. These AI web crawlers (Google, Perplexity, etc) are self-cannibalizing robots. They eat the goose that laid the golden egg for breakfast, and lose money doing it most of the time. If something isn't done to incentivize content creators again eventually there will be only walled-gardens and obsolete content left for the cannibals.
  - armchairhacker 1y ago
    AFAIK, currently creators get money while not charging for users because of ads. While I don’t blame creators for using ads now, I don’t think they’re a long-term solution. Ads are already blocked when people visit the site with ad blockers, which are becoming more popular. Obvious sponsored content may be blocked with the ads, and non-obvious sponsored content turns these “creators” into “shills” who are inauthentic and untrustworthy. Even without Google summaries, ad revenue may decrease over time as advertisers realize they aren’t effective or want more profit; even if it doesn’t, it’s my personal opinion that society should decrease the overall amount of ads. Not everyone creates only for money, the best only create for enough money to sustain themselves. A long-term solution is to expand art funding (e.g. creators apply for grants with their ideas and, if accepted, get paid a fixed rate to execute them) or UBI. Then media can be redistributed, remixed, etc. without impacting creators’ finances.

nazcan 1y ago

I think value is not proportional to bytes - an AI only needs to read a page once to add it to its model, and then served the effectively cached data many times.

Terretta 1y ago

Or, flip this, don't expect to get paid for pamphleteering?

novok 1y ago

The reason why that didn’t work was because regulations made micropayments too expensive, and the government wants it that way to keep control over the financial system.

OptionOfT 1y ago

> We're moving progressively in the direction of "pages can't be served for free anymore". Which, I don't think is a problem, and in fact I think it's something we should have addressed a long time ago. But it's done through a bait and switch. They serve the full article to Google, which allows Google to show you excerpts that you have to pay for. It would be better if Google shows something like PAYMENT REQUIRED on top, at least that way I know what I'm getting at.

mh- 1y ago
> They serve the full article to Google, which allows Google to show you excerpts that you have to pay for. I'm old enough to remember when that was grounds for getting your site removed from Google results - "cloaking" was against the rules. You couldn't return one result for Googlebot, and another for humans. No idea when they stopped doing that, but they obviously have let go of that principle.
- dspillett 1y ago
  I remember that too, along with high-profile punishments for sites that were keyword stuffing (IIRC a couple of decades ago BMW were completely unlisted for a time for this reason). I think it died largely because it became impossible top police with any reliability, and being strict about it would remove too much from Google's index because many sites are not easily indexable without them providing a “this is the version without all the extra round-trips for ad impressions and maybe a login needed” variant to common search engines. Applying the rule strictly would mean that sites implementing PoW tricks like Anubis to reduce unwanted bot traffic would not be included in the index if they serve to Google without the PoW step. I can't say I like that this has been legitimised even for the (arguably more common) deliberate bait & switch tricks is something I don't like, but (I think) I understand why the rule was allowed to slide.

chromatin 1y ago

402 Payment Required https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/402 https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/... Sadly development along these lines has not progressed. Yes, Google Cloud and other services may return it and require some manual human intervention, but I'd love to see _automatic payment negotiation_. I'm hopeful that instant-settlement options like Bitcoin Lightning payments could progress us past this. https://docs.lightning.engineering/the-lightning-network/l402 https://docs.lightning.engineering/the-lightning-network/l40... https://hackernoon.com/the-resurgence-of-http-402-in-the-age-of-agentic-ai https://hackernoon.com/the-resurgence-of-http-402-in-the-age...

makingstuffs 1y ago

As time passes I’m more certain in the belief that the internet will end up being a licensed system with insanely high barriers to entry which will stop your average dev from even being able to afford deploying a hobby project on it. Your idea of micro transacting web requests would play into it and probably end up with a system like Netflix where your ISP has access to a set of content creators to whom they grant ‘unlimited’ access as part of the service fee. I’d imagine that accessing any content creators which are not part of their package will either be blocked via a paywall (buy an addon to access X creators outside our network each month) or charged at an insane price per MB as is the case with mobile data. Obvious this is all super hypothetical but weirder stuff has happened in my lifetime

AlexandrB 1y ago

A scary observation in light of another front page article right now: https://news.ycombinator.com/item?id=44783566 https://news.ycombinator.com/item?id=44783566 If pages can't be served for free, all internet content is at the mercy of payment processors and their ideas of "brand safety".

inquirerGeneral 1y ago
[dead]
dspillett 1y ago
“Free” could have a number of meanings here. Free to the viewer, free to the hoster, free to the creator, etc… That content can't be served entirely for free doesn't mean that all content will require payment, and so is subject to issues with payment processors, just that some things may gravitate back to a model where it costs a small amount to host something (i.e. pay for home internet and host bits off that, or you might have VPS out there that runs tools and costs a few $ /yr or /month). I pay for resources to host my bits & bobs instead of relying on services provided in exchange for stalking the people looking at them, this is free for the viewer as they aren't even paying indirectly. Most things are paid for anyway, even if the person hosting it nor my looking at it are paying directly: adtech arseholes give services to people hosting content in exchange for the ability to stalk us and attempt to divert our attention. Very few sites/apps, other than play/hobby ones like mine or those from more actively privacy focused types, are free of that.
Taek 1y ago
That's already a deep problem for all of society. If we don't want that to be an ongoing issue, we need to make sure money is a neutral infrastructure. It doesn't just apply to the web, it applies to literally everything that we spend money on via a third party service. Which is... most everything these days.

bboygravity 1y ago

I get your thinking, but x.com is proof that simply making users pay (quite a lot) does not eliminate bots. The amount of "verified" paying "users" with a blue checkmark that are just total LLM bots is incredible on there. As long as spamming and DDOS'ing pays more than whatever the request costs, it will keep existing.

Saline9515 1y ago

Why would I pay for a page if I don't know if the content is what I asked for? How much are you going to pay? How much are you going to charge? This will end up in SEO hell, especially with AI-generated pages farming paid clicks.

adrian_b 1y ago

Your theory does not match the practice of Cloudflare. Whatever method is used by Cloudflare for detecting "threats" has nothing to do with consuming resources on the "protected" servers. The so-called "threats" are identified in users that may make a few accesses per day to a site, transferring perhaps a few kilobytes of useful data on the viewed pages (besides whatever amount of stupid scripts the site designer has implemented). So certainly Cloudflare does not meter the consumed resources. Moreover, Cloudflare preemptively annoys any user who accesses for the first time a site, having never consumed any resources, perhaps based on irrational profiling based on the used browser and operating system, and geographical location.