6 ms·

Everyone seems to be dismissing this without reading it. As a content publisher, I am very interested in any proposal that results in me getting residual paymen

Everyone seems to be dismissing this without reading it. As a content publisher, I am very interested in any proposal that results in me getting residual payments from AI scrapers. That would indeed be a new internet business model. This is also being attempted by RSL with their “Crawler Authentication Protocol” (https://rslstandard.org/guide/web-crawlers https://rslstandard.org/guide/web-crawlers) for demanding proof of licensing from scrapers and RSL Collective (https://rslcollective.org https://rslcollective.org) for providing the licensing itself. The missing piece there is the ability to detect scrapers with high accuracy without punishing regular browsing humans.

andy99 1y ago
> As a content publisher, I am very interested in any proposal that results in me getting residual payments from AI scrapers. So you use a public resource and presumably like the upside that comes with sharing on it, but you want to limit uses now that someone has found one that you don't like, so you're fine with degrading that public resource? You could always share things privately or behind a paywall if you don't want them available publicly. But people seem to want to have their cake and eat it too. I get why a hosting provider would want to limit crawlers to save bandwidth. The "creator" angle is just greed.
- velcrovan 1y ago
  > So you use a public resource and presumably like the upside that comes with sharing on it, but you want to limit uses now that someone has found one that you don't like, so you're fine with degrading that public resource? Can you say more? what is the "public resource" I'm using as a content publisher? In that role I see myself more as the provider of a public resource (my content), not a user.
  - carlosjobim 1y ago
    Maybe they mean the alphabet?
- pyrale 1y ago
  Your point is that pepole leaving some produce on the side of the road along with a money box should accept that some people loot all the produce in a 100km radius without paying, and resell it?
- beambot 1y ago
  You can put images up on a public website, but you still retain copyright & can control the images' use. This is no different from any kind of digital content...
  - isodev 1y ago
    And Cloudflare is trying to legitimise a “business model” resting on the assumption that it’s ok for AI vealers to scrape content in the first place. “Opt-in” is not in Silicon Valley vocabulary.
    - acdha 1y ago
      I read that differently: we’re starting from the current world where anyone publishing online is seeing expensive traffic volumes from companies whose goal is not to send user traffic back. Cloudflare seem to be arguing that site owners should easily be able to charge for agent access independently from regular user access, but since that is always under the control of the site owner it seems like the opposite of your “opt-in” complaint.
- computerdork 1y ago
  As a musician, yeah, don't agree with this. It'd be great to find a way practical way to get content creators compensated, because with the internet, almost all media is becoming free (or near free), and it's hard to survive. And, the idea being proposed is a version of what you're talking about (paywall). Not allowing crawlers to scrap your site unless they compensate you is a form of paywall, just for crawlers... ...But, I do agree with one aspect you're saying about greed, Cloudflare is probably being self serving in this. An open standard seems like it'd be a better solution than having one company control content payment.
oceanplexian 1y ago
I mean, you shouldn't need to read a wall of text to understand what a business model is. It should be front and center, communicated clearly, and easy to understand. If I was an investor, the lack of clarity after reading a few paragraphs would concern me.
- alex_suzuki 1y ago
  I read the entire article and I have no clue how this new business model will come to be. Something about filling holes im the cheese…
- velcrovan 1y ago
  It definitely isn't an aerated Linkedin post full of single-sentence paragraphs and emoji, but it's also certainly not a "wall of text". It's what we used to call a "blog post", it has very readable paragraphs broken up into short titled sections.
  - paxys 1y ago
    These days all content has to be in the form of a catchy 6 word tweet or 10 second reel to keep people's attention.
  - AuthAuth 1y ago
    There was no subway surfers video in the top how can we be expected to concentrate.
voxgen 1y ago
> without punishing regular browsing humans. As a content consumer, I'm also hoping to be part of the ecosystem. I already use Patreon a lot as "AdBlock absolution", but it doesn't fix the market dynamics. Major content platforms tend to stagnate or worsen over time, because they prefer to sell impressions to advertisers than a good product to consumers.
isodev 1y ago
I think as a publisher, you should be worried that Cloudflare is taking the first step towards the 2025 version of what “rights holders” have done to the movie and entertainment industry. Imagine trying to create and distribute a movie without the backing of your local distributor/broadcasting cartel. Only instead of gating the movie theatre, Cloudflare is the single access to all things web. Also, based in the US, so good luck producing content their government doesn’t like.
ElijahLynn 1y ago
I'm really excited by this post. I love the vision. especially the vision of using what is already known about the gaps and what is not known in having content created to fill those gaps. There are definitely some gaps in LLM knowledge.
_kidlike 1y ago
As an engineer who works at a medium-large publishing company, am also interested in this direction. I have fully read Cloudflare's proposal, as I have RSL, and L402. In its current state, RSL is not a full solution, but can be. The spec does not cover payments. Its basically just a new token type, which crawlers would register for to crawl your content, and predefined responses for more detailed states. But nothing around payments, yet. Cloudflare's solution is basically the same as L402. This is very sad for me, because their attempt is basically one of locking down the ecosystem in order to create a monopoly. If they were really about creating a fair internet where "pay per crawl" is possible (it already was with L402), they would openly support L402, implement it for their gateway, and try to push the industry towards it. But no, they just want the biggest piece of the pie; a pie that doesn't even exist yet.
- lfauve 1y ago
  It seems there are multiple initiatives. Have you also read this one? https://iabtechlab.com/announcing-content-monetization-protocols-comp-for-ai-working-group/ https://iabtechlab.com/announcing-content-monetization-proto... Maybe people have agendas, or maybe they have different perspectives that should be reconciled?
mayhemducks 1y ago
I like the IDEA of residual payments for content on paper. Brave browser tried this, but it didn't get enough traction from publishers of note to be viable. But I think, at least in theory, the idea of me as a consumer having some money attached to my eyeballs for the person/people who created the content I'm seeking out is a good one. Paywalls are annoying and awful, and unfortunately they seem to encourage data hording and selling practices that are unsavory. But I like the model of simply opening the browser, visiting a page, and the creators getting some royalty from that. The system we have now is too exploitative - it extracts more value that it creates.
criddell 1y ago
> As a content publisher It always bothers me a little when people describe themselves as a content publisher or content creator. Why content? Why not say you are a short film maker or writer or musician or videographer or journalist or painter or photographer or ...? In the past, your work has been merely content to the people who are looking for bits -- any bits -- to place between and around the stuff they really care about: ads. These days I suppose your work may be seen by some as grist for the AI mill. By calling your work content you are acceding to the idea that your work is only valuable in other contexts. It isn't! I bet it's something you made with care and thought. Or maybe I misread and you really are a content publisher. You're the person running the CMS who is looking for other people's work to monetize. If so, nevermind.
- ndriscoll 1y ago
  As a philosopher, I'd guess it's frequently because more concrete terms like "blogger" or "forum commenter" or "video game player" somehow lend even less credibility than the generic terminology.

andy99 1y ago

> As a content publisher, I am very interested in any proposal that results in me getting residual payments from AI scrapers. So you use a public resource and presumably like the upside that comes with sharing on it, but you want to limit uses now that someone has found one that you don't like, so you're fine with degrading that public resource? You could always share things privately or behind a paywall if you don't want them available publicly. But people seem to want to have their cake and eat it too. I get why a hosting provider would want to limit crawlers to save bandwidth. The "creator" angle is just greed.

velcrovan 1y ago
> So you use a public resource and presumably like the upside that comes with sharing on it, but you want to limit uses now that someone has found one that you don't like, so you're fine with degrading that public resource? Can you say more? what is the "public resource" I'm using as a content publisher? In that role I see myself more as the provider of a public resource (my content), not a user.
- carlosjobim 1y ago
  Maybe they mean the alphabet?
pyrale 1y ago
Your point is that pepole leaving some produce on the side of the road along with a money box should accept that some people loot all the produce in a 100km radius without paying, and resell it?
beambot 1y ago
You can put images up on a public website, but you still retain copyright & can control the images' use. This is no different from any kind of digital content...
- isodev 1y ago
  And Cloudflare is trying to legitimise a “business model” resting on the assumption that it’s ok for AI vealers to scrape content in the first place. “Opt-in” is not in Silicon Valley vocabulary.
  - acdha 1y ago
    I read that differently: we’re starting from the current world where anyone publishing online is seeing expensive traffic volumes from companies whose goal is not to send user traffic back. Cloudflare seem to be arguing that site owners should easily be able to charge for agent access independently from regular user access, but since that is always under the control of the site owner it seems like the opposite of your “opt-in” complaint.
computerdork 1y ago
As a musician, yeah, don't agree with this. It'd be great to find a way practical way to get content creators compensated, because with the internet, almost all media is becoming free (or near free), and it's hard to survive. And, the idea being proposed is a version of what you're talking about (paywall). Not allowing crawlers to scrap your site unless they compensate you is a form of paywall, just for crawlers... ...But, I do agree with one aspect you're saying about greed, Cloudflare is probably being self serving in this. An open standard seems like it'd be a better solution than having one company control content payment.

oceanplexian 1y ago

I mean, you shouldn't need to read a wall of text to understand what a business model is. It should be front and center, communicated clearly, and easy to understand. If I was an investor, the lack of clarity after reading a few paragraphs would concern me.

alex_suzuki 1y ago
I read the entire article and I have no clue how this new business model will come to be. Something about filling holes im the cheese…
velcrovan 1y ago
It definitely isn't an aerated Linkedin post full of single-sentence paragraphs and emoji, but it's also certainly not a "wall of text". It's what we used to call a "blog post", it has very readable paragraphs broken up into short titled sections.
- paxys 1y ago
  These days all content has to be in the form of a catchy 6 word tweet or 10 second reel to keep people's attention.
- AuthAuth 1y ago
  There was no subway surfers video in the top how can we be expected to concentrate.

voxgen 1y ago

> without punishing regular browsing humans. As a content consumer, I'm also hoping to be part of the ecosystem. I already use Patreon a lot as "AdBlock absolution", but it doesn't fix the market dynamics. Major content platforms tend to stagnate or worsen over time, because they prefer to sell impressions to advertisers than a good product to consumers.

isodev 1y ago

I think as a publisher, you should be worried that Cloudflare is taking the first step towards the 2025 version of what “rights holders” have done to the movie and entertainment industry. Imagine trying to create and distribute a movie without the backing of your local distributor/broadcasting cartel. Only instead of gating the movie theatre, Cloudflare is the single access to all things web. Also, based in the US, so good luck producing content their government doesn’t like.

ElijahLynn 1y ago

I'm really excited by this post. I love the vision. especially the vision of using what is already known about the gaps and what is not known in having content created to fill those gaps. There are definitely some gaps in LLM knowledge.

_kidlike 1y ago

As an engineer who works at a medium-large publishing company, am also interested in this direction. I have fully read Cloudflare's proposal, as I have RSL, and L402. In its current state, RSL is not a full solution, but can be. The spec does not cover payments. Its basically just a new token type, which crawlers would register for to crawl your content, and predefined responses for more detailed states. But nothing around payments, yet. Cloudflare's solution is basically the same as L402. This is very sad for me, because their attempt is basically one of locking down the ecosystem in order to create a monopoly. If they were really about creating a fair internet where "pay per crawl" is possible (it already was with L402), they would openly support L402, implement it for their gateway, and try to push the industry towards it. But no, they just want the biggest piece of the pie; a pie that doesn't even exist yet.

lfauve 1y ago
It seems there are multiple initiatives. Have you also read this one? https://iabtechlab.com/announcing-content-monetization-protocols-comp-for-ai-working-group/ https://iabtechlab.com/announcing-content-monetization-proto... Maybe people have agendas, or maybe they have different perspectives that should be reconciled?

mayhemducks 1y ago

I like the IDEA of residual payments for content on paper. Brave browser tried this, but it didn't get enough traction from publishers of note to be viable. But I think, at least in theory, the idea of me as a consumer having some money attached to my eyeballs for the person/people who created the content I'm seeking out is a good one. Paywalls are annoying and awful, and unfortunately they seem to encourage data hording and selling practices that are unsavory. But I like the model of simply opening the browser, visiting a page, and the creators getting some royalty from that. The system we have now is too exploitative - it extracts more value that it creates.

criddell 1y ago

> As a content publisher It always bothers me a little when people describe themselves as a content publisher or content creator. Why content? Why not say you are a short film maker or writer or musician or videographer or journalist or painter or photographer or ...? In the past, your work has been merely content to the people who are looking for bits -- any bits -- to place between and around the stuff they really care about: ads. These days I suppose your work may be seen by some as grist for the AI mill. By calling your work content you are acceding to the idea that your work is only valuable in other contexts. It isn't! I bet it's something you made with care and thought. Or maybe I misread and you really are a content publisher. You're the person running the CMS who is looking for other people's work to monetize. If so, nevermind.

ndriscoll 1y ago
As a philosopher, I'd guess it's frequently because more concrete terms like "blogger" or "forum commenter" or "video game player" somehow lend even less credibility than the generic terminology.