3 ms·
So you're suggesting as a website operator I should do nothing to resist and pay a large web hosting bill so that a company I've never heard of should benefit?
by trenchpilgrim 1y ago
So you're suggesting as a website operator I should do nothing to resist and pay a large web hosting bill so that a company I've never heard of should benefit? That is more directly harmful than this hypothetical third harm. What about my right to defend myself and my property?
- XenophileJKO 1y agoYou should block them, that is the ethical option.
- marginalia_nu 1y agoIf that worked this wouldn't be a discussion. Most of these misbehaved crawlers are either cloud hosted (with tens of thousands of IPs), using residential proxies (with tens of thousands of IPs) or straight up using a botnet (again with tens of thousands of IPs). None respect robots.txt and precious few even provide an identifiable user-agent string.
- trenchpilgrim 1y agoAs explained in the linked article, these bots have no identifiable properties by which to block them other than their scraping behavior. Some bots send each individual request from a separate origin.