3 ms·
If that worked this wouldn't be a discussion. Most of these misbehaved crawlers are either cloud hosted (with tens of thousands of IPs), using residential prox
by marginalia_nu 1y ago
If that worked this wouldn't be a discussion.
Most of these misbehaved crawlers are either cloud hosted (with tens of thousands of IPs), using residential proxies (with tens of thousands of IPs) or straight up using a botnet (again with tens of thousands of IPs). None respect robots.txt and precious few even provide an identifiable user-agent string.