The way to get around anti-bot stuff as a search engine crawler is to be upfront about what you are doing, as well as to respect robots.txt, and rate limits, an

The way to get around anti-bot stuff as a search engine crawler is to be upfront about what you are doing, as well as to respect robots.txt, and rate limits, and use a gratuitous crawl delay (1 sec is the bare minimum). You can register with cloudflare as a search engine crawler and they largely will let your traffic through. If you get blocked by individual sites, you can usually just email them explaining the situation and get unblocked. I'm having very little issues with this.