6 ms·
Cloudflare Radar: AI Insights
- chidog99 1y agoNothing better than a nice and clean dashboard
- h43z 1y agoI recently wanted to find out which company crawls the deepest. The openAI bot was the most thorough one, it followed 405 links [1]. [1] https://deep.43z.one https://deep.43z.one
- fleebee 1y agoNice stats! I've only had GPTBot reach depth 92 on my honeypot. I guess it's not as interesting.
- eric_khun 1y agoWondering if after this comment, you'll get more visit from those bots.
- tonyhart7 1y agosomebody troll that for level 1175 lul
- maxmcd 1y agoSuch a cool idea. I have https://chess.maxmcd.com/ https://chess.maxmcd.com/ and (before blocking most of them) many bots played thousands of moves deep. I remember bingbot was very active.
- jerrythegerbil 1y agoIf it’s been this way since February, how have AI crawlers not “caught up” yet? The internet is big, but it isn’t that big. I’d expect to see a sudden dropoff as they start re-checking content that hasn’t changed, with some sort of exponential backoff. Instead, my takeaway is that they are AI crawlers aren’t indexing to store in a way we’re used to with typical search engines, and unilaterally blocking these crawlers across the board would result in quite the “effect”.
- fresh_broccoli 1y agoI suppose these figures don't include the worst-behaving crawlers that hide their identity, e.g. by using residential proxies.
- aleyan 1y agoWhat an amazing set of data! The "Generative AI services popularity" [1] chart is surprising. ChatGPT is being #1 makes sense, but Character.AI being #2 is surprising, being ahead of Anthropic, Perplexity, and xAI. I suspect this data is strongly affected by the services DNS caching strategies. The other interesting chart is "Workers AI model popularity" [2]. `llama-3-8b-instruct` has been leading at 30% to 40% since April. That makes it hands the most popular weights available small "large language model". I would have expected Meta's `m2m100-1.2b` to be more used, as well as Alphabet's `Gemma 3 270M` starting to appear. People are likely using the most powerful model that fits on a CF worker. As shameless plug, for more popularity analysis, check out my "LLM Assistant Census" [3]. [1] https://radar.cloudflare.com/ai-insights#generative-ai-services-popularity https://radar.cloudflare.com/ai-insights#generative-ai-servi... [2] https://radar.cloudflare.com/ai-insights?dateRange=24w#workers-ai-model-popularity https://radar.cloudflare.com/ai-insights?dateRange=24w#worke... [3] https://aleyan.com/blog/2025-llm-assistant-census/ https://aleyan.com/blog/2025-llm-assistant-census/
- cj 1y agoWhy would DNS caching skew results? I don’t think Cloudflare is using DNS queries to compile the stats considering they have visibility into the full http requests for sites they proxy. Edit: Another comment mentions DNS queries. Did I miss something about how they’re compiling the stats?
- jcheng 1y agoThe heading says “Generative AI services popularity - Top 10 services based on 1.1.1.1 DNS resolver traffic”
- mmaunder 1y ago1.1.1.1 will see the query regardless of caching by upstream servers. Downstream and client caching probably averages out quite nicely with enough volume.
- deleted 1y ago
- secret-noun 1y ago> OpenAI > Verified via WebBotAuth: In Progress Feels like Cloudflare are positioning themselves as the gatekeepers of "good bots". The fact there is an "In Progress" state at all is telling: for everyone else, the answer is "No", but for OpenAI, the answer is "we're not doing it yet, but we've told CF that we plan to".
- evulhotdog 1y agoAmazon had a yes next to it.
- echelon 1y agoCloudFlare are going to tax the internet like Apple and Google tax smartphones. Ugh. On the one hand, I don't like AI bots consuming our traffic to build their proprietary products that they one day hope to put us out of business with. On the other hand, nobody asked Cloudflare to be the unelected leader of the internet. And I'm sure their policing and taxing will end here... God damnit, Internet. Can't we have nice open things? Every day in tech is starting to feel like geopolitical Game of Thrones. Kingdoms, winning wars, peasants...
- visarga 1y agoIf websites use Cloudflare to block AI bots the next wave of AI will rely on computer-use or browser-use to get in. Can you allow just humans and specific bots? I don't think so. The user problem is that web is borderline unusable because it is filled with ads, slop and trackers. Using AI makes it much better.
- throwaway1777 1y agoYou can if you have a stronger identity layer.
- esseph 1y agoWhich would absolutely suck
- troymc 1y agoMy main learning is that character.ai is consistently in the top four, along with ChatGPT (always #1) and Claude. I didn't even know it was in the running.
- emot 1y agosome related trends https://blog.cloudflare.com/crawlers-click-ai-bots-training/ https://blog.cloudflare.com/crawlers-click-ai-bots-training/
- pbd 1y agoThis data is incredibly valuable for both AI companies and publishers. CF gets unprecedented visibility into who's crawling what, when, and how much. Wouldn't be surprised if this becomes a premium product - 'pay for priority bot verification' or 'detailed crawl analytics.
- echelon 1y agoThis is going to be a huge growth lever for Cloudflare. They're going to milk OpenAI and the rest for everything they can.
- AlienRobot 1y agoDoes crawl-to-refer mean that for every 40k pages ClaudeBot crawls, only 1 outbound link is clicked from it?
- patrickhogan1 1y agoHow is it possible that training is much higher than search for use case?
- drexlspivey 1y agoIt needs to keep up with all the new content generated daily. Search on the other hand can be cached
- slig 1y ago>Top Browser & user agents > Firerox 3.8% This is sad. https://radar.cloudflare.com/adoption-and-usage https://radar.cloudflare.com/adoption-and-usage
- marcosdumay 1y agoHow much of this is because Cloudfare automatically classifying any Firefox as a bot and removing them from the statistics?
- gabeio 1y ago? I use firefox all of the time and I don’t believe I have been marked as a “bot”? I rarely hit website captchas/browser checks. Do you have anything to read that says otherwise?
- NicuCalcea 1y agoI use Firefox and have a VPN turned on most of the time, so I'm not sure which one's causing it, but I do occasionally get a Cloudflare page saying they've determined I'm a bot. Not captcha or anything, I'm just blocked from seeing the content.
- marcosdumay 1y agoWithout a VPN, you get Google captchas. Some times Google just decides you can not pass no matter what you do, but you still get the captchas.
- account42 1y agoI have no issues with Google captchas but CF just gives my Firefox install an endless spinner with no option except to contact them and provide them all the details that they couldn't collect automatically to "debug" the issue.
- reassess_blind 1y ago
- ChrisArchitect 1y agoRelated: Web Bot Auth https://news.ycombinator.com/item?id=45055452 https://news.ycombinator.com/item?id=45055452
- mmaunder 1y agoVery interesting data, particularly the AI rankings based on DNS requests. They appear to be off by one day because switching to a 4 week period, character AI is consistently #2 on weekends and Claude is #3 and they switch weekdays. But it’s shows the switch for Sunday and Monday. Probably a US time vs UTC issue.
- system2 1y agoThese AI companies popping up like mushrooms remind me of the .com bubble in the early 2000s.
- rvz 1y agoWe are now yet another month closer to the AI bubble collapsing. I am certain that Cloudflare will not be affected by an AI crash or AI winter at all.
- ec109685 1y agoIf I use Anthropic’s api for search, but then send user traffic directly to websites after showing the user the link, there’s no way for cloudflare to attribute that search to Anthropic. That makes the ratios of crawl to referrals shown suspect.
- kordlessagain 1y ago[flagged]
- egorfine 1y ago> Verified via WebBotAuth I sincerely hope this initiative fails and no one bends over for CloudFlare on this.
- jedahan 1y agoMy experience disagrees with the 'Respects robots.txt' column for most of the bots listed. Would love to see more details of how they determine that metric.
- o11c 1y agoAre you verifying the IP, or just blindly trusting the user agent?
- jedahan 1y agoGood question - I am just putting up robots.txt, and seeing little to no decrease in traffic. I have not tried verifying that server logs user agent corresponds to specific IP addresses. Do you have resources where all the AI bots post their list of IP addresses? Would be easier to just ban by IP completely. From what I've read these bots rotate and use residential blocks so I am not sure I can even see all of them.
- Anduia 1y agoAccording to that report, Grok has no respect whatsoever for anything
- shardullavekar 1y agowould be interesting to see if linkedin (and the likes who don't want to be crawled) signs up for the pay-per-crawl that CF may come up with.
- ccgreg 1y agoOne way that Cloudflare is gatekeeping is by declaring which bots are AI Bots. Common Crawl's CCBot is used for a lot of stuff -- it's an archive, there are more than 10,000 research papers citing common crawl, mostly not AI -- but Cloudflare deems CCBot to be an "AI Bot", and I suspect most website owners don't have any idea what the list of AI Bots is and how they were chosen.
- lyu07282 1y agowould be an obvious loophole if you could just use CC instead of paying cloudflare no?
- ccgreg 1y agoIt's a similar loophole as public libraries. When I was a kid, I read thousands of books from the library, without paying anyone anything. But as for the crawl loophole: CCBot obeys robots.txt, and CCBot also preserves all robots.txt and REPL signals so that downstream users can find out if a website intended to block them at crawl time.
- einrealist 1y agoPerhaps this data could provide a useful example for Apple and OpenAI in their defence against Elon's laughable lawsuit. It's funny how xAI is almost at the bottom.
- ashvardanian 1y agoThere's a nice write-up by Cloudflare from July covering some of those charts: https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar/ https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-r...
- verdverm 1y agoThe companies I avoid because they tried to charge my card even though I stopped using their service... Anthropic and OpenAI So interesting they are orders of magnitude worse than the others with the crawl:user-request ratio... noted
- gm678 1y agoI would have guessed that it's a minority, but less than 5% of web traffic being explicitly human-initiated is still a somewhat shocking statistic.
- yalogin 1y agoInstead of just rankings of AI chatbots, I wish there was a volume because I feel the volume skews heavily to the top
- vladak 1y agoNaive question maybe: do these AI companies crawl/ingest video/audio yet ? if yes, is that included in the stats ?
- jgalt212 1y agoHow is Googlebot not considered an AI bot? Googlebot feeds all the AI snippets and zero-click internet. Googlebot is an AI bot.
- gasmankohl 1y agothis feels like Cloudflare is no longer solely on the "serving the website owner's" side anymore
- PeterStuer 1y agoCloudflare is positioning themselves to be the Internet's tax collector.
- Ilikeruby 1y agoDead internet theory starting to not be a theory anymore
- rixed 1y agoRegarding WebBotauth, I've just skimmed the doc but I don't understand why didn't they have used TLS client certificate instead of this new header? Is it because it's then easier for Cloudflare to strip that header than to do TLS termination?