Just want to say THANK YOU for this insight. It never occurred to me, and I just checked out one of the big sites that I scrape for a side project and, lo and

Just want to say THANK YOU for this insight. It never occurred to me, and I just checked out one of the big sites that I scrape for a side project and, lo and behold, you are 100% correct. Found their origin in Censys in about 30 seconds and I've never been able to crunch through their pages more easily. To others out there who explore this: As with all scraping, be gentle! If you start pounding on someone's origin server directly, you're much more likely to be noticed than if you're pounding on something behind a CloudFlare cache. Set rate limits, scrape during off-peak hours, etc. Be a good scraping citizen.