I do, some third party search crawlers are just badly programmed, and after you get burned a bunch of times you just want to deny anyone who isn't one of the ma

I do, some third party search crawlers are just badly programmed, and after you get burned a bunch of times you just want to deny anyone who isn't one of the main players. I think they are basically startups with a lot of money to spend on crawl compute, but who haven't really figured out their crawl engine, and it can go wild on your site. You also have bots that seem to be credential stuffing, bots that seem to be content scrapping (stuff with same typos shows up elsewhere after their visits, really obvious on new / fresh articles, bots that seem to be exploring for copyright claims, rando bots (maybe comment sentiment analysis for stock trading) etc. Google is much more welcome by comparison.