3 ms·
While as a developer I find this super interesting, as a system administrator this makes me cringe. We don't have a lot of resources for servers, and I end up s
by tesin 9y ago
While as a developer I find this super interesting, as a system administrator this makes me cringe. We don't have a lot of resources for servers, and I end up spending a disproportionate amount of time banning IPs from bots running poorly configured tools like this, which aren't rate limited and crush websites.
I'm grateful that "Obey robots.txt" is listed as part of it's standard behavior. If only scrapers cared enough to use it as well.
- codedokode 9y agoSuch crawler should not be difficult to ban by looking at stats - if there are many requests per IP per unit of time, or many requests from data center IPs, or many requests from Linux browsers, it is likely bots and you can ban them (you can ban whole data center to be sure).
- superasn 9y agoI've found that mod_evasive[1] works particularly well in these situation and helped us a lot dealing with it (though I'm not a sysadmin and I'm sure there are better tools to deal with it). But for someone who is just a webmaster, I'd recommend using it for a quick dirty fix for such hassles. [1] https://www.digitalocean.com/community/tutorials/how-to-protect-against-dos-and-ddos-with-mod_evasive-for-apache-on-centos-7 https://www.digitalocean.com/community/tutorials/how-to-prot...