2 ms·
How does Cloudflare know whether a bot is scraping for AI purposes or any other? Or are they blocking all bots but basically adding AI to the feature because AI
by altdataseller 2y ago
How does Cloudflare know whether a bot is scraping for AI purposes or any other? Or are they blocking all bots but basically adding AI to the feature because AI is the cool thing right now?
- rsynnott 2y agoI assume a correct robots.txt, plus attempt to kill off robots.txt evaders; if you could do that reliably (the second bit is the tricky bit, of course), you could selectively exclude any bots you want.