3 ms·

> For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will

by tekacs 2mo ago

> For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default. It's kind of exhausting seeing Cloudflare playing both sides of the arms race. I just can't imagine bringing myself to use their technology to build agents and build AI products when they're also doing things like this. > This also lines up the incentive model we want to foster. Losing trusted status across the more than 20% of web domains that sit behind Cloudflare is a deterrent with teeth. Trust becomes something you can carry with you, and something you can lose. And even more so, LLM language aside, fun and fascinating to see them flagrantly calling out their position here as if it's a positive.

deleted 2mo ago
[deleted]
usef- 2mo ago
How are they playing both sides? I thought their scraping products were also about having it behave and not take down systems
- tekacs 2mo ago
  I mean that they're telling developers that they should use Cloudflare's platform to build agents, the kind of agents that would go across the web and act on behalf of users... but then they're also the ones blocking those requests. This always engenders a solid amount of distaste from me, because much like Google and Chrome, it creates the incentive for you to treat yourself better than others. Especially coupled with the trust stuff. Of course, Cloudflare is always going to trust their own platform.
  - stingraycharles 2mo ago
    I thought the issue people have with AI scrapers is the ones that DDoS sites to scrape everything for training purposes, rather than the ones that interactively query specific content to support an active conversation?
    - PunchyHamster 2mo ago
      First one is problem because it brings no traffic back and occasionally DDoSes the site Second one is problem because it DDoSes the site. "Just" queries for hundred thousand people (if you happened to be good source for that bit of knowledge) that don't bring actual people to your site is also a problem
      - inigyou 2mo ago
        First one is not occasional, it is the anonymous global adversary probably ddosing your site right now. It never relents.
    - tekacs 2mo ago
      If you read the article, you'll notice that they're explicitly making sure to block the ones that interactively query specific content too.
  - samrus 2mo ago
    There are ways to use those agents ethically. Cloudflare isnt forcing you to be a dick with your agents. Infact the blocking side encourages you to use them ethically. I dont see the problem.
- nirui 2mo ago
  > not take down systems "Block on pages with ads" is probably about preventing the AI crawlers from clicking on the ads which maybe considered cheating by the ad company. If you want to prevent "bot attacks", maybe the "Block" option will do the trick. But of course, to do all that you need to put some trust on Cloudflare, because they're the one identifying the bots from normal users. For me, as someone who's hosting a Gitea instance behind Cloudflare, I have a Configuration Rule set that says: if the client is trying to access a URL that is beyond certain length limit, then trigger "Browser Integrity Check" and "I’m Under Attack", a.k.a stricter security checks. The match expression of the rule looked something like this: ( len(http.request.uri) > !!!!!SET LENGTH LIMIT!!!!! and not lower(http.request.uri.path) contains ".git/" and not lower(http.request.uri.path) contains "api/" ) (The `!!!!!SET LENGTH LIMIT!!!!!` is an integer of the length limit you wanted to set) This rule alone basically blocked all abusive bot traffic to almost zero for my site (https://i.imgur.com/LaOjjvV.png https://i.imgur.com/LaOjjvV.png, see the traffic drop around 10 clock and Cloudflare mitigation kicks in). But of course, you need to figure out your own rules based on the characteristic of the website. Also, you can be more creative: for example, my actual rule is more complex than that, it also checks to see if a cookie is not set, and only triggers when all condition are met: ( len(http.request.uri) > !!!!!SET LENGTH LIMIT!!!!! and not lower(http.request.uri.path) contains ".git/" and not lower(http.request.uri.path) contains "api/" and not http.cookie wildcard "*!!!!!COOKIE NAME!!!!!=!!!!!COOKIE VALUE!!!!!*" ) then, as part two of that rule, I have a Response Header Transform Rules that says: ( len(http.request.uri) <= !!!!!SET LENGTH LIMIT!!!!! and not http.cookie contains "!!!!!COOKIE NAME!!!!!=!!!!!COOKIE VALUE!!!!!" ) and if this Response Header Transform Rules is triggered, it sets the cookie `!!!!!COOKIE NAME!!!!!=!!!!!COOKIE VALUE!!!!!`. (Note: `!!!!!COOKIE NAME!!!!!` and `!!!!!COOKIE VALUE!!!!!` are the variables you need to customize) When you put the two rules together, it forces clients to access "shallow" (short URL) pages first as an user would normally do, before they can access "deeper" (long URL) content hosted on the site without triggering more strict security checks. If that makes sense. Also, don't forget the cookie basically also dug a hole in the security setting. So it's really a balance between avoid annoying the user and protect your site. You need to be smart and be flexible about it, otherwise your users will just leave.
  - 2mo ago
- inigyou 2mo ago
  Their scraping products are mostly about paying them money to not be blocked.
  - siva7 2mo ago
    So Google has to pay Cloudflare 10B$ to get Googlebot moved to their default Allowlist otherwise Google would loose access to a significant portion of the internet? Genius move.
    - inigyou 2mo ago
      Yes literally this. Every large corporation gatekeeper does this sort of thing, it's why we have to militantly push for decentralisation by avoiding these corporations. It's not morally different from Google forcing your site to be AI-scrapeable or be kicked off Google Search. It's why Apple won't let you publish an app without paying them 30% of your total revenue. It's why Microsoft threatened to ban Steam (they later decided not to) so they could get all those games onto the Microsoft Store, and in response Valve ported gaming to Linux.
    - schainks 2mo ago
      Wait until you find out what Google is paying Reddit…
colechristensen 2mo ago
I see Cloudflare as trying to forge the appropriate path ahead. Neither allowing the free-for-all nor trying to block everything isn't playing both sides, it's the path straight down the middle. Providing tools for producers and consumers to do things with permission and compensation.

by tekacs 2mo ago

deleted 2mo ago

[deleted]

usef- 2mo ago

How are they playing both sides? I thought their scraping products were also about having it behave and not take down systems

tekacs 2mo ago
I mean that they're telling developers that they should use Cloudflare's platform to build agents, the kind of agents that would go across the web and act on behalf of users... but then they're also the ones blocking those requests. This always engenders a solid amount of distaste from me, because much like Google and Chrome, it creates the incentive for you to treat yourself better than others. Especially coupled with the trust stuff. Of course, Cloudflare is always going to trust their own platform.
- stingraycharles 2mo ago
  I thought the issue people have with AI scrapers is the ones that DDoS sites to scrape everything for training purposes, rather than the ones that interactively query specific content to support an active conversation?
  - PunchyHamster 2mo ago
    First one is problem because it brings no traffic back and occasionally DDoSes the site Second one is problem because it DDoSes the site. "Just" queries for hundred thousand people (if you happened to be good source for that bit of knowledge) that don't bring actual people to your site is also a problem
    - inigyou 2mo ago
      First one is not occasional, it is the anonymous global adversary probably ddosing your site right now. It never relents.
  - tekacs 2mo ago
    If you read the article, you'll notice that they're explicitly making sure to block the ones that interactively query specific content too.
- samrus 2mo ago
  There are ways to use those agents ethically. Cloudflare isnt forcing you to be a dick with your agents. Infact the blocking side encourages you to use them ethically. I dont see the problem.
nirui 2mo ago
> not take down systems "Block on pages with ads" is probably about preventing the AI crawlers from clicking on the ads which maybe considered cheating by the ad company. If you want to prevent "bot attacks", maybe the "Block" option will do the trick. But of course, to do all that you need to put some trust on Cloudflare, because they're the one identifying the bots from normal users. For me, as someone who's hosting a Gitea instance behind Cloudflare, I have a Configuration Rule set that says: if the client is trying to access a URL that is beyond certain length limit, then trigger "Browser Integrity Check" and "I’m Under Attack", a.k.a stricter security checks. The match expression of the rule looked something like this: ( len(http.request.uri) > !!!!!SET LENGTH LIMIT!!!!! and not lower(http.request.uri.path) contains ".git/" and not lower(http.request.uri.path) contains "api/" ) (The `!!!!!SET LENGTH LIMIT!!!!!` is an integer of the length limit you wanted to set) This rule alone basically blocked all abusive bot traffic to almost zero for my site (https://i.imgur.com/LaOjjvV.png https://i.imgur.com/LaOjjvV.png, see the traffic drop around 10 clock and Cloudflare mitigation kicks in). But of course, you need to figure out your own rules based on the characteristic of the website. Also, you can be more creative: for example, my actual rule is more complex than that, it also checks to see if a cookie is not set, and only triggers when all condition are met: ( len(http.request.uri) > !!!!!SET LENGTH LIMIT!!!!! and not lower(http.request.uri.path) contains ".git/" and not lower(http.request.uri.path) contains "api/" and not http.cookie wildcard "*!!!!!COOKIE NAME!!!!!=!!!!!COOKIE VALUE!!!!!*" ) then, as part two of that rule, I have a Response Header Transform Rules that says: ( len(http.request.uri) <= !!!!!SET LENGTH LIMIT!!!!! and not http.cookie contains "!!!!!COOKIE NAME!!!!!=!!!!!COOKIE VALUE!!!!!" ) and if this Response Header Transform Rules is triggered, it sets the cookie `!!!!!COOKIE NAME!!!!!=!!!!!COOKIE VALUE!!!!!`. (Note: `!!!!!COOKIE NAME!!!!!` and `!!!!!COOKIE VALUE!!!!!` are the variables you need to customize) When you put the two rules together, it forces clients to access "shallow" (short URL) pages first as an user would normally do, before they can access "deeper" (long URL) content hosted on the site without triggering more strict security checks. If that makes sense. Also, don't forget the cookie basically also dug a hole in the security setting. So it's really a balance between avoid annoying the user and protect your site. You need to be smart and be flexible about it, otherwise your users will just leave.
- 2mo ago
inigyou 2mo ago
Their scraping products are mostly about paying them money to not be blocked.
- siva7 2mo ago
  So Google has to pay Cloudflare 10B$ to get Googlebot moved to their default Allowlist otherwise Google would loose access to a significant portion of the internet? Genius move.
  - inigyou 2mo ago
    Yes literally this. Every large corporation gatekeeper does this sort of thing, it's why we have to militantly push for decentralisation by avoiding these corporations. It's not morally different from Google forcing your site to be AI-scrapeable or be kicked off Google Search. It's why Apple won't let you publish an app without paying them 30% of your total revenue. It's why Microsoft threatened to ban Steam (they later decided not to) so they could get all those games onto the Microsoft Store, and in response Valve ported gaming to Linux.
  - schainks 2mo ago
    Wait until you find out what Google is paying Reddit…

colechristensen 2mo ago

I see Cloudflare as trying to forge the appropriate path ahead. Neither allowing the free-for-all nor trying to block everything isn't playing both sides, it's the path straight down the middle. Providing tools for producers and consumers to do things with permission and compensation.