4 ms·

We also use Fail2Ban like a poor man's Web Application Firewall - for a Rails app you know you'll never have legit traffic to anything 'cgi' or 'php' or 'exe',

We also use Fail2Ban like a poor man's Web Application Firewall - for a Rails app you know you'll never have legit traffic to anything 'cgi' or 'php' or 'exe', so crafting a couple of regexps to match those and then ban them works well. We use CloudFlare so we feed the IP bans into CloudFlare's firewall through their API as well as local IPTables. (We also use CloudFlare's WAF - we like redundancy in our paranoia.) If you had more demanding requirements you could look at mod_security for Apache or Nginx. We also implement rate-limiting using Fail2Ban, which helps to trip the dumber automated scans. Every time Fail2Ban blocks an IP address, it emails our admin team with the matching log entries and a whois on the IP address, which we then eyeball to make sure there's nothing weird going on. TBH, easily 90% of the IP addresses we block got caught looking for 'wp-admin/admin.php', which is probably a good reason to never use WordPress.

_Chief 8y ago
I like this approach. Do you permaban the IPs? because a ton of ISPs rotate IPs across users
- semanticist 8y ago
  CloudFlare has a maximum number of firewall rules, based on the types of paid accounts you have. Every couple of months we purge the list when it gets close to being full. 99% of IP addresses we block are from outside our target market - we're a very local/regional company - so we don't really have to worry much about blocking legitimate customers. Very rarely our support team will get a request from someone that's been blocked and we'll unban the IP address, or set it to a JS/Captcha challenge at CloudFlare.
inopinatus 8y ago
Just be aware when crafting those regexes that user-generated URLs (e.g. slug-based content paths, or even just query-string params if one is very careless) can thereby be used to build a denial of service attack against internal infrastructure, third-party integrations/webhooks, and even regular customers. Reminds me that back in the day we used to be able to train email sanitizing appliances for false positives by sending them EICAR or GTUBE with spoofed headers.
- semanticist 8y ago
  We have definitely run into this, albeit only very very rarely. It's not viable for someone to build a usable DoS against us as an attack: our offices are on a whitelist and our set up prevents a viable attack against our internal infrastructure. But yeah, like every tool, there's potential risks/down sides/things to watch for. There's no silver bullets!
eli 8y ago
Are these bots causing problems like consuming excess resources? If not it doesn't seem worth the effort to ban them. These days I only block bots when they're e.g. filling out forms or posting junk comment that a human has to moderate.
- PeterisP 8y ago
  Some kid running an automated vulnerability scan can be automatically detected and blocked this way, which can protect you if that scan would have succeeded otherwise. Of course, the proper way would be just to scan all that yourself and actually fix the problems, but that may be too difficult in some cases.
- yani 8y ago
  Scanning bots consume resources and traffic. I pay for both. They increase my bill so it makes sense to restrict them.
  - eli 8y ago
    Having gone down this path before, I think time is better spent making your site more efficient so it doesn't matter. Developer time and site complexity are much more important commodities than bandwidth fees.
    - yani 8y ago
      I usually leave it to the devops to deal with it. If it has to be a developer on my team to do it then you are right, I will spend their time elsewhere.
- raverbashing 8y ago
  It seems you never had to deal with the amount of crap bots throw at every minor website out there. If you don't want your logs flooded with vuln. scanning and referrer spam, use fail2ban or a waf.
  - 8y ago
yani 8y ago
We also use fail2ban. It serves us good. When adding a new rule, we dont apply it but collect all the matched requests and ips. After 1 week, we review the log and if no customer traffic was captured, the rule is applied. We ban only for 30m. In my field, web, it takes 2m for a bot to disappear once its requests have been blocked. We dont use permanent bans as we havent had the need for them
belorn 8y ago
Fail2ban is for all practical purposes resource depletion protection against bots that pose no threat but still generate work for the server. For this purpose you do sacrifice a bit of I/O performance, but hopefully won't get those spikes that brings things down. It is not a replacement for the actually security of keeping things updated and having good passwords/client certificates.

_Chief 8y ago

I like this approach. Do you permaban the IPs? because a ton of ISPs rotate IPs across users

semanticist 8y ago
CloudFlare has a maximum number of firewall rules, based on the types of paid accounts you have. Every couple of months we purge the list when it gets close to being full. 99% of IP addresses we block are from outside our target market - we're a very local/regional company - so we don't really have to worry much about blocking legitimate customers. Very rarely our support team will get a request from someone that's been blocked and we'll unban the IP address, or set it to a JS/Captcha challenge at CloudFlare.

inopinatus 8y ago

Just be aware when crafting those regexes that user-generated URLs (e.g. slug-based content paths, or even just query-string params if one is very careless) can thereby be used to build a denial of service attack against internal infrastructure, third-party integrations/webhooks, and even regular customers. Reminds me that back in the day we used to be able to train email sanitizing appliances for false positives by sending them EICAR or GTUBE with spoofed headers.

semanticist 8y ago
We have definitely run into this, albeit only very very rarely. It's not viable for someone to build a usable DoS against us as an attack: our offices are on a whitelist and our set up prevents a viable attack against our internal infrastructure. But yeah, like every tool, there's potential risks/down sides/things to watch for. There's no silver bullets!

eli 8y ago

Are these bots causing problems like consuming excess resources? If not it doesn't seem worth the effort to ban them. These days I only block bots when they're e.g. filling out forms or posting junk comment that a human has to moderate.

PeterisP 8y ago
Some kid running an automated vulnerability scan can be automatically detected and blocked this way, which can protect you if that scan would have succeeded otherwise. Of course, the proper way would be just to scan all that yourself and actually fix the problems, but that may be too difficult in some cases.
yani 8y ago
Scanning bots consume resources and traffic. I pay for both. They increase my bill so it makes sense to restrict them.
- eli 8y ago
  Having gone down this path before, I think time is better spent making your site more efficient so it doesn't matter. Developer time and site complexity are much more important commodities than bandwidth fees.
  - yani 8y ago
    I usually leave it to the devops to deal with it. If it has to be a developer on my team to do it then you are right, I will spend their time elsewhere.
raverbashing 8y ago
It seems you never had to deal with the amount of crap bots throw at every minor website out there. If you don't want your logs flooded with vuln. scanning and referrer spam, use fail2ban or a waf.
- 8y ago

yani 8y ago

We also use fail2ban. It serves us good. When adding a new rule, we dont apply it but collect all the matched requests and ips. After 1 week, we review the log and if no customer traffic was captured, the rule is applied. We ban only for 30m. In my field, web, it takes 2m for a bot to disappear once its requests have been blocked. We dont use permanent bans as we havent had the need for them

belorn 8y ago

Fail2ban is for all practical purposes resource depletion protection against bots that pose no threat but still generate work for the server. For this purpose you do sacrifice a bit of I/O performance, but hopefully won't get those spikes that brings things down. It is not a replacement for the actually security of keeping things updated and having good passwords/client certificates.