4 ms·
You should block them, that is the ethical option.
by XenophileJKO 1y ago
You should block them, that is the ethical option.
- marginalia_nu 1y agoIf that worked this wouldn't be a discussion. Most of these misbehaved crawlers are either cloud hosted (with tens of thousands of IPs), using residential proxies (with tens of thousands of IPs) or straight up using a botnet (again with tens of thousands of IPs). None respect robots.txt and precious few even provide an identifiable user-agent string.
- trenchpilgrim 1y agoAs explained in the linked article, these bots have no identifiable properties by which to block them other than their scraping behavior. Some bots send each individual request from a separate origin.