3 ms·

This problem is only going to get worse. for my thegreatestbooks.org site i used to just get indexed/scraped by google and bing. now it's like 50+ AI bots scrap

by wtf242 2y ago

This problem is only going to get worse. for my thegreatestbooks.org site i used to just get indexed/scraped by google and bing. now it's like 50+ AI bots scraping my entire site just so they can train a LLM to answer questions my site answers without having a user ever visit my site. I just checked cloudflare and in the past 24 hours I've had 1.2 million bot/automated requests

sct202 2y ago
There's a new setting in Cloudflare to block AI/scraper bots. https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click https://blog.cloudflare.com/declaring-your-aindependence-blo...
- graeme 2y ago
  Anyone have any experience with this? Is there nothing but upside in blocking these bots
  - account42 2y ago
    Considering it's Buttflare, enabling it probably also means blocking random users. But of course that's not Buttflare's problem because it's not enabled by default.