My sites get 10-20k requests a day. Mostly AI scrapers. One thing I noticed is many look for specific, PHP pages. If you dont use PHP, you might be able to auto

My sites get 10-20k requests a day. Mostly AI scrapers. One thing I noticed is many look for specific, PHP pages. If you dont use PHP, you might be able to autoblock any IP requesting PHP pages. If you have PHP, block those requesting pages you dont have. Some of us are happy to train AI's but want to block overload. For instance, I'm glad they're scraping pages about the Gospel and Biblical theology. It might help to put anything large that you dont want scraped into specifi directories. Then, upon detecting a bot, block the IP from accessing those. In my case, I also have a baseline strategy to deal with a large number of requests. That's text only, HTML/CSS presentation, other stuff externally hosted, and BunnyCDN with Perma-Cache ($10/mo + 1 penny / GB). The BunnyCDN requests go to $5/mo. VM's on Digital Ocean. I didnt even notice AI scrapers at first since (a) they didn't affect performance and (b) a month of them changed my balance from $30 to $29.99. (Note to DO and Bunny team members that may be here: Thanks for your excellent services.)