3 ms·
I think the problem is some IPs just straight-up always get CAPTCHAs from Cloudflare even if one’s a good netizen, respect robots.txt, not crawl at ridiculous r
by oefrha 5y ago
I think the problem is some IPs just straight-up always get CAPTCHAs from Cloudflare even if one’s a good netizen, respect robots.txt, not crawl at ridiculous rate, and not lie in the user agent. One reason is shared IP, which disproportionally affects people from third world countries as their ISPs don’t have enough IPv4 for everyone; but it also happened mysteriously to at least one dedicated IP I used in the past. Your confrontational tone is rather unfortunate, and the problem of course is that you don’t guarantee anything even if the user has done nothing wrong, as is manifest from the choice of the phrase “starting point”.
- ehutch79 5y agoThen the problem has nothing to do with your crawling.