4 ms·
Cloudflare adds option to block AI scrapers and crawlers
- ppcland 2y ago[dead]
- Daunk 2y agoDidn't they add this last year?
- SpaceManNabs 2y agoGlad to see the repost. I am going to check if I have this enabled. didn't know this feature existed.
- bakugo 2y agoI'm guessing the option to block AI bots using WAF rules has been available since last year, but this simple toggle on/off option is recent (though, all it does is create a WAF rule for you, which seemingly counts towards the WAF rule limit)
- xbar 2y agoI thought so. I wish they had added it 5 years ago.
- jdon 2y agoCloudflare's blog post from last year [1] contains more details. [1] https://blog.cloudflare.com/ai-bots/ https://blog.cloudflare.com/ai-bots/
- ipaddr 2y agoCloudflare is great at blocking users legitimate or not.
- tomschlick 2y agoThere is always going to be false positives. Even more so with a tech/privacy minded userbase which uses ad blockers, disables JS, etc.
- jgalt212 2y agoI personally have had more issues with Google Captchas than Cloudflare Turnstile when using the dreaded Firefox / Linux combo.
- bornfreddy 2y agoOf course. Google has every incentive to make life difficult for ff users, while CF doesn't.
- altdataseller 2y agoHow does Cloudflare know whether a bot is scraping for AI purposes or any other? Or are they blocking all bots but basically adding AI to the feature because AI is the cool thing right now?
- rsynnott 2y agoI assume a correct robots.txt, plus attempt to kill off robots.txt evaders; if you could do that reliably (the second bit is the tricky bit, of course), you could selectively exclude any bots you want.
- chollida1 2y agowhat is the difference between a scraper or crawler and an "AI scraper/crawler"? And how does cloudflare differentiate between them?
- deleted 2y ago[deleted]
- twelve40 2y agoThey'll just get you through common crawl or something else innocent-looking. The urge to milk everyone's content to the last penny is just too great.