Even then, I've run into issues with scraping several sites for the reverse image search engine I operate. Luckily, in most cases I have been able to get in tou

Even then, I've run into issues with scraping several sites for the reverse image search engine I operate. Luckily, in most cases I have been able to get in touch with the people running those sites to get a rule added for my IPs to allow them through. That's not scalable though, and limits where I can scrape/crawl from. Even something as simple as checking a site for updates every hour or two tends to get blocked after a few times. TBH, one of the only things I have found which helps is lying in the user agent and copying CF cookies. Luckily, I haven't had to play with that for a few months due to whitelisting, so not sure it it would still help. Things change rapidly.