3 ms·

FWIW all of those IP's are in FireHol 30 day netset [1] from the FireHol Repo [2] They are known abusers and safe to null route if you happen to run your own m

FWIW all of those IP's are in FireHol 30 day netset [1] from the FireHol Repo [2] They are known abusers and safe to null route if you happen to run your own mail servers, blogs, etc... wc -l firehol_abusers* 7260 firehol_abusers_1d.netset 184010 firehol_abusers_30d.netset # first grep for any IP's you care about and then: for Ip in $(cat firehol_abusers_30d.netset);do ip route add blackhole "${Ip}" 2>/dev/null;done If that cuts down on the noise then create a startup script to do this on reboot. The first time this should be done from a web console vs. ssh in case the list contains your own IP address or gateway. Why ip route vs ipset or iptables? Far less CPU load than netfilter. A more generic way to keep many bots out is to redirect anything other than HTTP/2.0 to a password protected listener. This will block Google bots as they still do not support HTTP/2.0 but will also block a majority of the problematic bots. [1] - https://github.com/firehol/blocklist-ipsets/blob/master/firehol_abusers_30d.netset https://github.com/firehol/blocklist-ipsets/blob/master/fire... [2] - https://github.com/firehol/blocklist-ipsets https://github.com/firehol/blocklist-ipsets

tranxen 4y ago
There are almost 200k IP in that file. Would it not cause a performance issue for your OS to have that many routes to lookup ? Even if it is still way better than using iptable.
- LinuxBender 4y ago
  Would it not cause a performance issue for your OS to have that many routes to lookup? Not for a blog. If this were a high performance computing node that needed extremely low latency CPU instruction performance then I would find another way but route enumeration at least in Linux and BSD are incredibly fast and efficient. I can't speak for other operating systems. One method to test this would be to just start null routing more of those netsets not in a startup script and run load testing tools against the blog to see where it starts getting slower and to keep an eye on memory usage. The bigger files to test with are: 2475418 firehol_proxies.netset 2480723 firehol_anonymous.netset
  - lovehashbrowns 4y ago
    For millions of IPs I put them all in redis as a key, and wrote some Lua code for nginx. The lua code would do a key look up in redis, and store the result of that lookup in cache. If the IP was in redis as a key, nginx would let the request time out. No error of any kind. I also had to update that IP blacklist daily so that’s also why I chose redis. Probably better ways to do it nowadays but that’s what I did like 8 years ago and it’s still one of my favorite solutions ever. Also part of why nginx and redis will forever be two of my favorite technologies alongside Linux.

tranxen 4y ago

There are almost 200k IP in that file. Would it not cause a performance issue for your OS to have that many routes to lookup ? Even if it is still way better than using iptable.

LinuxBender 4y ago
Would it not cause a performance issue for your OS to have that many routes to lookup? Not for a blog. If this were a high performance computing node that needed extremely low latency CPU instruction performance then I would find another way but route enumeration at least in Linux and BSD are incredibly fast and efficient. I can't speak for other operating systems. One method to test this would be to just start null routing more of those netsets not in a startup script and run load testing tools against the blog to see where it starts getting slower and to keep an eye on memory usage. The bigger files to test with are: 2475418 firehol_proxies.netset 2480723 firehol_anonymous.netset
- lovehashbrowns 4y ago
  For millions of IPs I put them all in redis as a key, and wrote some Lua code for nginx. The lua code would do a key look up in redis, and store the result of that lookup in cache. If the IP was in redis as a key, nginx would let the request time out. No error of any kind. I also had to update that IP blacklist daily so that’s also why I chose redis. Probably better ways to do it nowadays but that’s what I did like 8 years ago and it’s still one of my favorite solutions ever. Also part of why nginx and redis will forever be two of my favorite technologies alongside Linux.