3 ms·
I wonder if a service exists where IPs known for crawling are reported and reputation of such IPs is tracked for others to use and ban / ratelimit by default
by temp008 2y ago
I wonder if a service exists where IPs known for crawling are reported and reputation of such IPs is tracked for others to use and ban / ratelimit by default
- mrweasel 2y agoThe problem with that approach is that you'll quickly add large swaths of IPs belonging to cloud service provides, such as AWS. We already know that AWS, Azure, GCP and Alibaba are part of the problem, so we can technically just rate-limit them already. I believe that all of them publish their IP ranges. Google also publish the IP ranges for GoogleBot I believe, and Bing probably does the same, so we can then whitelist those IPs and still have sites appear in searches. My issue is that the burden is again placed on everyone else, not the people/companies who are causing the problem. It's crazy to me to think about how much needless capacity is built into the internet to deal with crawlers. The resource waste is just insane.