3 ms·

robots.txt is purely advisory and many of the people who code a crawler which could get in a state where it hits you hundreds of times a minute might not be gre

by patio11 13y ago

robots.txt is purely advisory and many of the people who code a crawler which could get in a state where it hits you hundreds of times a minute might not be great about actually respecting it. If it is a crawler, though, it is probably operating from a single C block, likely on AWS or what have you. You'd probably just drop them at the firewall. Simple, no fuss. I'm not a sysadmin, and I have nothing but respect for you and your team with the following recommendation: if it isn't obviously how to implement "drop that C block at the firewall" then find yourself a competent contract sysadmin, pay him for one hour of work today, then pay him for 10 hours of work over the next few weeks beefing up your infrastructure. It will be cheap at the price. If you're absolutely determined to do it yourself the magic word is "iptables" and may God have mercy on your soul, because iptables is the most obtuse piece of software I've ever had the misfortune of trying to configure. In the alternative, if you're actually facing a botnet, the expected behavior is lots of requests from widely distributed IPs. You'd be able to verify this with your access logs (which record IPs and timestamps) and a 10 or 20 line shell script to put a histogram of unique requests per IP. If it's a few requests from thousands of IPs per hour, then you have a bit of a challenge to address it on your end of things. My first totally-not-a-sysadmin advice would be to dial down dynamic content within the app for non-logged-in users and start aggressively caching pages, which might or might not be enough for you to weather the storm. Best of luck and skill.

panabee 13y ago
ok thanks for the suggestions! one other data point: we seem to have a ton of sleeping mysql connections that continue to crop up even after we try killing them. is this symptomatic of a botnet attack?
- patio11 13y ago
  I am insufficiently skilled at MySQL to diagnose probable reasons for that, but think it is unlikely that it would be uniquely triggered by a botnet. Instead, I'm betting that it is a problem with the application (PHP, maybe?) which is opening one new connection per page view and failing to close it, either because the application is improperly coded or because the application might be dying due to lack of resources (caused by your traffic spike) and not clean up after itself in the event of death.
- thaumaturgy 13y ago
  If you're really actually in a bind, drop me a line (email's in profile). I'll give you a hand. If you're looking for advice on how to mitigate this yourself: - apachetop is a useful tool for this sort of thing, and since the activity is occurring on a single site, it should be easy to use. (Assuming of course you use apache.) - If it's an actual botnet, you'll need to pick some honeypot URLs -- links that regular customers aren't hitting but the botnet is -- and then use something like Fail2Ban to pipe accesses to those URLs into iptables bans. - If it's a crawler, it should be from a limited set of IPs and a quick iptables rule addition will fix you up. For reference, here are two easy-to-use rules: iptables -A INPUT -s <source_ip_address> -j DROP iptables -A INPUT -m iprange --src-range <source_ip_range> -j DROP The first rule will add a "drop" command to your INPUT table for any traffic coming from <source_ip_address>. The second rule will allow you to quickly drop a range of IP addresses; <source_ip_range> should look something like "10.0.0.1-10.0.1.255", and "iprange" is a literal.
  - obituary_latte 13y ago
    Just wanted to say: that honeypot URL -> fail2ban idea is awesome. Thanks for sharing.
    - euroclydon 13y ago
      Just be careful not to ban Googlebot. It's pretty good at following new links.
      - obituary_latte 13y ago
        Good point. It seems the bot is also good at minding robots.txt so that might be an option. Could also have the honeypot script to do a quick whois on the ip before passing along to fail2ban.

by patio11 13y ago

panabee 13y ago

ok thanks for the suggestions! one other data point: we seem to have a ton of sleeping mysql connections that continue to crop up even after we try killing them. is this symptomatic of a botnet attack?

patio11 13y ago
I am insufficiently skilled at MySQL to diagnose probable reasons for that, but think it is unlikely that it would be uniquely triggered by a botnet. Instead, I'm betting that it is a problem with the application (PHP, maybe?) which is opening one new connection per page view and failing to close it, either because the application is improperly coded or because the application might be dying due to lack of resources (caused by your traffic spike) and not clean up after itself in the event of death.
thaumaturgy 13y ago
If you're really actually in a bind, drop me a line (email's in profile). I'll give you a hand. If you're looking for advice on how to mitigate this yourself: - apachetop is a useful tool for this sort of thing, and since the activity is occurring on a single site, it should be easy to use. (Assuming of course you use apache.) - If it's an actual botnet, you'll need to pick some honeypot URLs -- links that regular customers aren't hitting but the botnet is -- and then use something like Fail2Ban to pipe accesses to those URLs into iptables bans. - If it's a crawler, it should be from a limited set of IPs and a quick iptables rule addition will fix you up. For reference, here are two easy-to-use rules: iptables -A INPUT -s <source_ip_address> -j DROP iptables -A INPUT -m iprange --src-range <source_ip_range> -j DROP The first rule will add a "drop" command to your INPUT table for any traffic coming from <source_ip_address>. The second rule will allow you to quickly drop a range of IP addresses; <source_ip_range> should look something like "10.0.0.1-10.0.1.255", and "iprange" is a literal.
- obituary_latte 13y ago
  Just wanted to say: that honeypot URL -> fail2ban idea is awesome. Thanks for sharing.
  - euroclydon 13y ago
    Just be careful not to ban Googlebot. It's pretty good at following new links.
    - obituary_latte 13y ago
      Good point. It seems the bot is also good at minding robots.txt so that might be an option. Could also have the honeypot script to do a quick whois on the ip before passing along to fail2ban.