4 ms·
Are these bots causing problems like consuming excess resources? If not it doesn't seem worth the effort to ban them. These days I only block bots when they're
by eli 9y ago
Are these bots causing problems like consuming excess resources? If not it doesn't seem worth the effort to ban them.
These days I only block bots when they're e.g. filling out forms or posting junk comment that a human has to moderate.
- PeterisP 9y agoSome kid running an automated vulnerability scan can be automatically detected and blocked this way, which can protect you if that scan would have succeeded otherwise. Of course, the proper way would be just to scan all that yourself and actually fix the problems, but that may be too difficult in some cases.
- yani 9y agoScanning bots consume resources and traffic. I pay for both. They increase my bill so it makes sense to restrict them.
- eli 9y agoHaving gone down this path before, I think time is better spent making your site more efficient so it doesn't matter. Developer time and site complexity are much more important commodities than bandwidth fees.
- yani 9y agoI usually leave it to the devops to deal with it. If it has to be a developer on my team to do it then you are right, I will spend their time elsewhere.
- raverbashing 9y agoIt seems you never had to deal with the amount of crap bots throw at every minor website out there. If you don't want your logs flooded with vuln. scanning and referrer spam, use fail2ban or a waf.
- eli 9y agoI have, in fact, been doing this for years. Why should I care about having pristine logs?
- raverbashing 9y agoYeah, who cares about the extra disk space, the extra overhead, the waste of resources, filtering through noise to understand some issues with the app, right? Isn't it lovely when every other referrer is trying to promote some online pharmacy or testing to see if you have a wp vulnerability? Sure you can filter it, but I'd rather deal with garbage at the source.
- eli 9y agoA lesson I learned the hard way: if random bots are causing significant resourcing issues, then you need to address that directly. And if their impact isn't significant, why are you spending time on it? The disk space my log files consume is trivial.
- semanticist 9y agoWhile it's mostly bots trying to attack Wordpress, it's also blocked a few people spidering us aggressively (probably competitors) and other people trying attacks. Once it's scripted through a Chef cookbook, it's applied to all of our infrastructure with almost no effort, including any new boxes we launch. The biggest effort is running a script every few months to clear out the CloudFlare firewall, and I could probably stick that in a cron job, really. Our technical team is very small and the company entirely dependent on our website and our reputation, so we take multiple approaches to securing the site and servers. Each little thing we add helps my boss sleep better, and that alone is probably worth the initial effort. I definitely wouldn't do what some people are suggesting, manually reviewing logs/fail2ban matches and blocking IP addresses by hand. That's a waste of effort - and would probably be a full time job, too.