I run a crawler and I dunno if I agree with this description. Usually anti-bot measures are in place for a reason, either to protect some paywalled content fro

I run a crawler and I dunno if I agree with this description. Usually anti-bot measures are in place for a reason, either to protect some paywalled content from scraping, or to prevent bots from getting at forms and causing a mess. Few webmasters have cloudflare (or similar) anti-bot measures turned up higher than it needs to be (because it 100% will annoy people), and content that isn't a high-value target for blackhat bot activity is rarely difficult to crawl.