4 ms·

Thanks. We spent half the day with them and are awaiting a confirmation from their network guys that this is, indeed, a botnet attack. Unfortunately, the networ

by panabee 13y ago

Thanks. We spent half the day with them and are awaiting a confirmation from their network guys that this is, indeed, a botnet attack. Unfortunately, the network guys aren't in until the morning -- only the general support people are -- prompting the post on HN. So much for the 24/7 support they claim ... We also suspect it might be an aggressive search engine. Our next step is to pull some IP data. If it turns out to be a search engine, what can we do beyond editing robots.txt? If it's not a search engine, how can we confirm a botnet attack ourselves. We don't want to wait for these network guys to wake up and drink their morning coffee. Thanks again!

jonah 13y ago
If the majority of the requests are coming from a certain set of IPs, configure your server to immediately drop those requests to reduce the load on your app.
patio11 13y ago
robots.txt is purely advisory and many of the people who code a crawler which could get in a state where it hits you hundreds of times a minute might not be great about actually respecting it. If it is a crawler, though, it is probably operating from a single C block, likely on AWS or what have you. You'd probably just drop them at the firewall. Simple, no fuss. I'm not a sysadmin, and I have nothing but respect for you and your team with the following recommendation: if it isn't obviously how to implement "drop that C block at the firewall" then find yourself a competent contract sysadmin, pay him for one hour of work today, then pay him for 10 hours of work over the next few weeks beefing up your infrastructure. It will be cheap at the price. If you're absolutely determined to do it yourself the magic word is "iptables" and may God have mercy on your soul, because iptables is the most obtuse piece of software I've ever had the misfortune of trying to configure. In the alternative, if you're actually facing a botnet, the expected behavior is lots of requests from widely distributed IPs. You'd be able to verify this with your access logs (which record IPs and timestamps) and a 10 or 20 line shell script to put a histogram of unique requests per IP. If it's a few requests from thousands of IPs per hour, then you have a bit of a challenge to address it on your end of things. My first totally-not-a-sysadmin advice would be to dial down dynamic content within the app for non-logged-in users and start aggressively caching pages, which might or might not be enough for you to weather the storm. Best of luck and skill.
- panabee 13y ago
  ok thanks for the suggestions! one other data point: we seem to have a ton of sleeping mysql connections that continue to crop up even after we try killing them. is this symptomatic of a botnet attack?
  - patio11 13y ago
    I am insufficiently skilled at MySQL to diagnose probable reasons for that, but think it is unlikely that it would be uniquely triggered by a botnet. Instead, I'm betting that it is a problem with the application (PHP, maybe?) which is opening one new connection per page view and failing to close it, either because the application is improperly coded or because the application might be dying due to lack of resources (caused by your traffic spike) and not clean up after itself in the event of death.
  - thaumaturgy 13y ago
    If you're really actually in a bind, drop me a line (email's in profile). I'll give you a hand. If you're looking for advice on how to mitigate this yourself: - apachetop is a useful tool for this sort of thing, and since the activity is occurring on a single site, it should be easy to use. (Assuming of course you use apache.) - If it's an actual botnet, you'll need to pick some honeypot URLs -- links that regular customers aren't hitting but the botnet is -- and then use something like Fail2Ban to pipe accesses to those URLs into iptables bans. - If it's a crawler, it should be from a limited set of IPs and a quick iptables rule addition will fix you up. For reference, here are two easy-to-use rules: iptables -A INPUT -s <source_ip_address> -j DROP iptables -A INPUT -m iprange --src-range <source_ip_range> -j DROP The first rule will add a "drop" command to your INPUT table for any traffic coming from <source_ip_address>. The second rule will allow you to quickly drop a range of IP addresses; <source_ip_range> should look something like "10.0.0.1-10.0.1.255", and "iprange" is a literal.
    - obituary_latte 13y ago
      Just wanted to say: that honeypot URL -> fail2ban idea is awesome. Thanks for sharing.
abcd_f 13y ago
> the network guys aren't in until the morning That's not a very good ISP. You should consider switching away from them as soon as you can.
awalton 13y ago
> So much for the 24/7 support they claim Step 1: Get a new provider.

by panabee 13y ago

jonah 13y ago

If the majority of the requests are coming from a certain set of IPs, configure your server to immediately drop those requests to reduce the load on your app.

patio11 13y ago

robots.txt is purely advisory and many of the people who code a crawler which could get in a state where it hits you hundreds of times a minute might not be great about actually respecting it. If it is a crawler, though, it is probably operating from a single C block, likely on AWS or what have you. You'd probably just drop them at the firewall. Simple, no fuss. I'm not a sysadmin, and I have nothing but respect for you and your team with the following recommendation: if it isn't obviously how to implement "drop that C block at the firewall" then find yourself a competent contract sysadmin, pay him for one hour of work today, then pay him for 10 hours of work over the next few weeks beefing up your infrastructure. It will be cheap at the price. If you're absolutely determined to do it yourself the magic word is "iptables" and may God have mercy on your soul, because iptables is the most obtuse piece of software I've ever had the misfortune of trying to configure. In the alternative, if you're actually facing a botnet, the expected behavior is lots of requests from widely distributed IPs. You'd be able to verify this with your access logs (which record IPs and timestamps) and a 10 or 20 line shell script to put a histogram of unique requests per IP. If it's a few requests from thousands of IPs per hour, then you have a bit of a challenge to address it on your end of things. My first totally-not-a-sysadmin advice would be to dial down dynamic content within the app for non-logged-in users and start aggressively caching pages, which might or might not be enough for you to weather the storm. Best of luck and skill.

panabee 13y ago
ok thanks for the suggestions! one other data point: we seem to have a ton of sleeping mysql connections that continue to crop up even after we try killing them. is this symptomatic of a botnet attack?
- patio11 13y ago
  I am insufficiently skilled at MySQL to diagnose probable reasons for that, but think it is unlikely that it would be uniquely triggered by a botnet. Instead, I'm betting that it is a problem with the application (PHP, maybe?) which is opening one new connection per page view and failing to close it, either because the application is improperly coded or because the application might be dying due to lack of resources (caused by your traffic spike) and not clean up after itself in the event of death.
- thaumaturgy 13y ago
  If you're really actually in a bind, drop me a line (email's in profile). I'll give you a hand. If you're looking for advice on how to mitigate this yourself: - apachetop is a useful tool for this sort of thing, and since the activity is occurring on a single site, it should be easy to use. (Assuming of course you use apache.) - If it's an actual botnet, you'll need to pick some honeypot URLs -- links that regular customers aren't hitting but the botnet is -- and then use something like Fail2Ban to pipe accesses to those URLs into iptables bans. - If it's a crawler, it should be from a limited set of IPs and a quick iptables rule addition will fix you up. For reference, here are two easy-to-use rules: iptables -A INPUT -s <source_ip_address> -j DROP iptables -A INPUT -m iprange --src-range <source_ip_range> -j DROP The first rule will add a "drop" command to your INPUT table for any traffic coming from <source_ip_address>. The second rule will allow you to quickly drop a range of IP addresses; <source_ip_range> should look something like "10.0.0.1-10.0.1.255", and "iprange" is a literal.
  - obituary_latte 13y ago
    Just wanted to say: that honeypot URL -> fail2ban idea is awesome. Thanks for sharing.

abcd_f 13y ago

> the network guys aren't in until the morning That's not a very good ISP. You should consider switching away from them as soon as you can.

awalton 13y ago

> So much for the 24/7 support they claim Step 1: Get a new provider.