3 ms·

ok thanks for the suggestions! one other data point: we seem to have a ton of sleeping mysql connections that continue to crop up even after we try killing them

by panabee 13y ago

ok thanks for the suggestions! one other data point: we seem to have a ton of sleeping mysql connections that continue to crop up even after we try killing them. is this symptomatic of a botnet attack?

patio11 13y ago
I am insufficiently skilled at MySQL to diagnose probable reasons for that, but think it is unlikely that it would be uniquely triggered by a botnet. Instead, I'm betting that it is a problem with the application (PHP, maybe?) which is opening one new connection per page view and failing to close it, either because the application is improperly coded or because the application might be dying due to lack of resources (caused by your traffic spike) and not clean up after itself in the event of death.
thaumaturgy 13y ago
If you're really actually in a bind, drop me a line (email's in profile). I'll give you a hand. If you're looking for advice on how to mitigate this yourself: - apachetop is a useful tool for this sort of thing, and since the activity is occurring on a single site, it should be easy to use. (Assuming of course you use apache.) - If it's an actual botnet, you'll need to pick some honeypot URLs -- links that regular customers aren't hitting but the botnet is -- and then use something like Fail2Ban to pipe accesses to those URLs into iptables bans. - If it's a crawler, it should be from a limited set of IPs and a quick iptables rule addition will fix you up. For reference, here are two easy-to-use rules: iptables -A INPUT -s <source_ip_address> -j DROP iptables -A INPUT -m iprange --src-range <source_ip_range> -j DROP The first rule will add a "drop" command to your INPUT table for any traffic coming from <source_ip_address>. The second rule will allow you to quickly drop a range of IP addresses; <source_ip_range> should look something like "10.0.0.1-10.0.1.255", and "iprange" is a literal.
- obituary_latte 13y ago
  Just wanted to say: that honeypot URL -> fail2ban idea is awesome. Thanks for sharing.
  - euroclydon 13y ago
    Just be careful not to ban Googlebot. It's pretty good at following new links.
    - obituary_latte 13y ago
      Good point. It seems the bot is also good at minding robots.txt so that might be an option. Could also have the honeypot script to do a quick whois on the ip before passing along to fail2ban.

by panabee 13y ago

patio11 13y ago

I am insufficiently skilled at MySQL to diagnose probable reasons for that, but think it is unlikely that it would be uniquely triggered by a botnet. Instead, I'm betting that it is a problem with the application (PHP, maybe?) which is opening one new connection per page view and failing to close it, either because the application is improperly coded or because the application might be dying due to lack of resources (caused by your traffic spike) and not clean up after itself in the event of death.

thaumaturgy 13y ago

If you're really actually in a bind, drop me a line (email's in profile). I'll give you a hand. If you're looking for advice on how to mitigate this yourself: - apachetop is a useful tool for this sort of thing, and since the activity is occurring on a single site, it should be easy to use. (Assuming of course you use apache.) - If it's an actual botnet, you'll need to pick some honeypot URLs -- links that regular customers aren't hitting but the botnet is -- and then use something like Fail2Ban to pipe accesses to those URLs into iptables bans. - If it's a crawler, it should be from a limited set of IPs and a quick iptables rule addition will fix you up. For reference, here are two easy-to-use rules: iptables -A INPUT -s <source_ip_address> -j DROP iptables -A INPUT -m iprange --src-range <source_ip_range> -j DROP The first rule will add a "drop" command to your INPUT table for any traffic coming from <source_ip_address>. The second rule will allow you to quickly drop a range of IP addresses; <source_ip_range> should look something like "10.0.0.1-10.0.1.255", and "iprange" is a literal.

obituary_latte 13y ago
Just wanted to say: that honeypot URL -> fail2ban idea is awesome. Thanks for sharing.
- euroclydon 13y ago
  Just be careful not to ban Googlebot. It's pretty good at following new links.
  - obituary_latte 13y ago
    Good point. It seems the bot is also good at minding robots.txt so that might be an option. Could also have the honeypot script to do a quick whois on the ip before passing along to fail2ban.