11 ms·
Cloudflare's new AI traffic options for customers
- ray_v 3mo agoSo, in summary: still the honors system. Got it. thanks.
- zx8080 3mo agoWhat's the "honors system"?
- strictnein 3mo agoRobots.txt
- ray_v 3mo agoBingo!
- willy_k 3mo agoAn honor system is a system without any (explicit) external enforcement of rules. For example an unnattended fruit stand, where people are trusted to be honorable and leave payment for the fruit they take.
- graeme 3mo agoHas there been any update on the pay per crawl program?
- simonw 3mo agoThe big news here is that Googlebot will be blocked from September 15th onwards by one the "block training" policies, because Google use the same crawler infrastructure for their search index AND for training Gemini: > Another change that will apply on September 15 is that multi-purpose crawlers (specifically those that combine Search with Training) will be allowed/blocked according to all of their behaviors, in line with our call for transparency for website owners. Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service).
- jofzar 3mo agoWe had googlebot blast a random customer system and almost cause an outage, this is when I first learnt that google will use it for AI training also. It's honestly kind of frustrating also because you then search on it and theres (was) nothing on how you are meant to "correctly" tell google to fuck off, and not use it like that.
- 20k 3mo agoGoogle's web scraping functionality has been acting as a ddos for more than two decades. I've seen literally hundreds of reports of them attacking websites and taking them down, where there's nothing you can do but accept the traffic, or get delisted This is unfortunately nothing new. There's no correct way to tell them to fuck off, they do not care, and they never will do. People have even taken them to court over this
- weird-eye-issue 3mo agoIf a site cannot handle traffic from the real Googlebot that is a serious issue with the site itself since it's actually pretty conservative Also I should note there are lots of fake Googlebots...
- remus 3mo agoIndeed, I've got a site which gets a lot of bot traffic and google bot is pretty sensible compared to a lot of other mainstream bots.
- weird-eye-issue 3mo agoYes, it really does not make that many requests. In fact lots of site owners struggle with having it not crawl and index their site enough
- 20k 3mo agoIt is mostly, but it doesn't take a lot of googling to find sites getting ridiculous amounts of traffic from googlebot on google IPs. Its one of the most common complaints about google's search indexing
- tekacs 3mo ago> For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default. It's kind of exhausting seeing Cloudflare playing both sides of the arms race. I just can't imagine bringing myself to use their technology to build agents and build AI products when they're also doing things like this. > This also lines up the incentive model we want to foster. Losing trusted status across the more than 20% of web domains that sit behind Cloudflare is a deterrent with teeth. Trust becomes something you can carry with you, and something you can lose. And even more so, LLM language aside, fun and fascinating to see them flagrantly calling out their position here as if it's a positive.
- deleted 3mo ago[deleted]
- usef- 3mo agoHow are they playing both sides? I thought their scraping products were also about having it behave and not take down systems
- tekacs 3mo agoI mean that they're telling developers that they should use Cloudflare's platform to build agents, the kind of agents that would go across the web and act on behalf of users... but then they're also the ones blocking those requests. This always engenders a solid amount of distaste from me, because much like Google and Chrome, it creates the incentive for you to treat yourself better than others. Especially coupled with the trust stuff. Of course, Cloudflare is always going to trust their own platform.
- stingraycharles 3mo agoI thought the issue people have with AI scrapers is the ones that DDoS sites to scrape everything for training purposes, rather than the ones that interactively query specific content to support an active conversation?
- zzzeek 3mo agothis is annoying, it makes a big deal about "Back when we announced pay-per-crawl"... I want pay-per-crawl. I clicked the link for it a year ago, got presented with a "request access" button, I "requested access" and obviously since I'm nobody I heard absolutely nothing. Now they're touting the link again, I checked, still that same "request access" button. I have no idea if anyone even has access to this feature. I don't care about all this other stuff, I want the AI crawlers to pay me cash. Because boy do those fuckers want to crawl me. I'll gladly double the size of my gerrit/jenkins servers to keep up with the load if these stupid bots want to pay to crawl every jenkins build artifact and every changeset source file on the server, as they really seem to want to do.
- youre-wrong3 3mo ago[dead]
- arjie 3mo agoThis is fine so long as it’s easy for me to turn off. I just don’t want to accidentally lose all AI traffic one day.
- dmortin 3mo agoDo you actually get any signifcant AI traffic today?
- arjie 3mo agoYeah, maybe like 10:1 bot to human traffic. I'm just a rando blogger so I have principally machine readers - which is not a thing I mind. But I do use Cloudflare everywhere and I could just as well have PEBKACked a setting somewhere to block the traffic. Amusingly, I used to have like 100x the readers in the 2000s posting just as inane takes. I've been left behind by the refinement of the craft!
- PeterStuer 3mo agoMost of the internet unfortunatly ploinks fown a 'free' service and never even looks at what it defaults to. E.g., quite a lott of cloudflare "protected" sites block their rss feeds from being read by machine.
- inigyou 3mo agoThis is deliberate from Cloudflare. It wants to block Google's access to as much of the internet as possible.
- fc417fc802 3mo agoPlease consider installing one of the many PoW schemes such as anubis rather than use these cloudflare "features". I increasingly encounter outright blocks rather than any sort of captcha when visiting cloudflare "protected" sites. Each individual site isn't particularly important to me but it's depressing to watch the process unfold like this. You really are choosing to erode the core basis of the internet if you go this route.
- gruez 3mo ago>I increasingly encounter outright blocks rather than any sort of captcha when visiting cloudflare "protected" sites. ??? Unless you're browsing around with the googlebot user agent string, you should be getting turnstile challanges at most, not blocks. And if you're getting a turnstile challenge it's unclear how it's different than an anubis challenge. If you're outright blocked, it's probably a site decision (eg. block all VPNs or block everyone not from a given country) rather than cloudflare's.
- gucci-on-fleek 3mo ago> And if you're getting a turnstile challenge it's unclear how it's different than an anubis challenge. You are guaranteed to pass an Anubis challenge eventually [0], whereas it's possible to get stuck forever in an infinitely-looping Turnstile challenge. > If you're outright blocked, it's probably a site decision (eg. block all VPNs or block everyone not from a given country) rather than cloudflare's. Cloudflare blocks legitimate users itself sometimes [1]. [0]: Unless you run into a bug, but Anubis is open source, so you can always submit a patch upstream. I've done this myself, and I can confirm that it's relatively straightforward. [1]: https://news.ycombinator.com/item?id=43329320 https://news.ycombinator.com/item?id=43329320
- gruez 3mo ago>You are guaranteed to pass an Anubis challenge eventually [0] That's a double edged sword because bots will eventually get through too, and unlike humans, their time is dirt cheap. >Cloudflare blocks legitimate users itself sometimes [1]. I never ran into this issue despite using seemingly maximally suspicious configs like tor browser. I can't say the same for some other vendors.
- holografix 3mo agoWhat’s the end goal for Cloudflare and the web here? I don’t think ADOG (anthropic, deepmind, openai, google) is going to pay to crawl. What would force their hand? It’s more likely they’ll strike undisclosed agreements with major sources of discussion like reddit etc. That’s not to say getting new information as a way of context-providing is not going to happen but that’s not scraping.
- sandeepkd 3mo agoI get a mixed feeling about all this. Cloudflare is unilaterally making all these decisions which impact the whole internet traffic flow. Taking the lead is one thing, however decisions like this should have the direct involvement of Internet Engineering Task Force (IETF) to account for all stakeholders, otherwise we run into the situation of a fragmented internet
- usef- 3mo agoI'm curious if this is just to pressure Google into separating their crawlers
- sandeepkd 3mo agoThis is more of a way to create a unauthorized toll tax on highway. There is a problem indeed, however if the proposed direction by cloud flare is to solve it or benefit out of it is a debatable topic.
- inigyou 3mo agoThat wouldn't make any sense from anyone's perspective. They would just pretend to separate them and not.
- colechristensen 3mo agoAre they unilateral? Maybe the defaults? For anyone competent they're settings freely chosen.
- jerf 3mo ago
- paul7986 3mo ago[dead]
- Fizz43 3mo ago>So, instead of defining a bot primarily as “AI” or not, our updated approach to classification will ask deeper questions about bot or agent behavior: What are they doing on my site? What are they storing? And how will they reshare my content? I dont get this. The question is are they a bot or a human. It doesnt matter what they are doing I dont want bots on my site.
- akersten 3mo ago> I dont get this. The question is are they a bot or a human. It doesnt matter what they are doing I dont want bots on my site. Do you want your site to be discoverable by a search engine? (How do you think that occurs?)
- Terr_ 3mo agoLet's also spare a moment for accessibility issues. For example, is it that wrong for a blind person to invoke a tool that describes a picture when there's no alt-text? Or something which describes/transcribes audio for the deaf? Where do we draw the line between a personal-bot and a custom browser?
- Imustaskforhelp 3mo agoCould there be a proper committee/association which can have all good faith search engines (which work for the purpose of search rather than AI related) which could show their proper IP/networks and cloudflare could have an option which supports all search but not any AI botnets. That being said, the issue with it right now is that Google uses the same crawler for both AI training and search, so this step puts a small pressure on google nonetheless to hopefully split them.
- inigyou 3mo agoWhy not. You need a reason. A browser is just a bot that renders pages.
- sneak 3mo agoWebsite operators don’t lose anything when people download the content from their website and use it. There is no technical mechanism whereby it is actually possible to allow people to read your webpage and not use it for other things. You can’t give responses that say “this is ok for indexing but not for training”. Anyone trying to sell you this sort of technology is lying.
- noduerme 3mo agoI wonder if this has anything to do with the cf bug that stripped all POST data from requests to a SPA I manage for 4-5 hours last week. That was a real good time, figuring out that it wasn't trying to show challenges or anything. Default setting for any web app protection from cloudflare should always be "off" unless you're under attack, and then who knows what settings will or won't break your configuration.
- TekMol 3mo agoSo is it possible to say "No bots except Google, OpenAI, Grok, Claude and Perplexity"? As far as I can tell, Google is the only one sending me visitors. And the other big AI players might do so in the future. Another option would be "No anonymous bots". So at least if a bot would want to crawl my site, they would have to identify themselves. Since the rise of the AI bots, I am getting hurt badly with insane amounts of requests from residential IPs that mimic real humans. The only difference being they don't make me any money. Only produce costs. By the way, how is the situation over at Amazon's Cloudfront? Do they offer something that helps? Anyone here with them?
- nicbou 3mo agoGoogle is actively working on not sending you visitors anymore.
- jwr 3mo agoI find it unsettling that we are willingly outsourcing the decision on who can access our sites to an increasingly dominant corporate entity. The reasoning behind this is also flawed: blocking "bots" and "AI" means that our AI agents working for us are unable to do their work for us, because of knee-jerk bot-blocks.
- timpera 3mo agoBlocking bots should not be the default behavior, it should be opt-in. Is Cloudflare trying to play both sides in order to secure a fee on every pay-to-crawl in the future?
- guyn 3mo agoI tried this, blocking AI training blocked the Google search bots and cut my traffic in half. I would not recommend.
- edifierxuhao 3mo ago[flagged]
- julian-vix 3mo ago[flagged]
- fllkfsalkdsfds 3mo ago[dead]
- leros 3mo agoIs anyone using something like this? I'm skeptical of scrapers paying and, if paying, paying enough to make it worthwhile.
- buildwrangler 3mo agoI do appreciate that cloudflare is one of the few platforms offering options to web sites on how they want to engage with AI and even monetize traffic coming from it.