6 ms·
Ask HN: A botnet may be attacking our site right now, how should we respond?
It seems like our ecommerce site is under attack from a botnet, causing our site to go down. We're receiving 50-100 requests per second (far surpasses normal traffic). Some requests are for outdated URLs not even normally accessible from the site.
Here's IP data related to requests: https://gist.github.com/panabee/9595411. Can post more if more is needed for diagnosis.
Questions:
1) How do we confirm if the site is under attack?
2) If the site is under attack, how can we ward off the attack and prevent future ones?
3) We seem to have a ton of sleeping MySQL connections that continue to crop up even after we try killing them. Is this symptomatic of a botnet attack?
We appreciate any help or guidance anyone can offer.
Thanks!
- BorisMelnik 13y agosomething that low it could just be a spam program testing to see if you have certain pages available. for instance some programs look for "signup.php" or "register.html" and if = exists will proceed with a registration bot. contact your host. sounds like normal normal shit.
- patio11 13y agoFirst step: get in touch with your hosting provider. They're better at this than you are, and may have suggestions or knobs they can twiddle on their end. You might just be getting crawled by an overly aggressive search engine, by the way. (Probably not one of the big guys.) From your perspective that is isomorphic to being hit with a botnet, but the nice thing about that is the search engine won't attempt to get around the mitigation steps you put in place. (Well, typically.)
- panabee 13y agoThanks. We spent half the day with them and are awaiting a confirmation from their network guys that this is, indeed, a botnet attack. Unfortunately, the network guys aren't in until the morning -- only the general support people are -- prompting the post on HN. So much for the 24/7 support they claim ... We also suspect it might be an aggressive search engine. Our next step is to pull some IP data. If it turns out to be a search engine, what can we do beyond editing robots.txt? If it's not a search engine, how can we confirm a botnet attack ourselves. We don't want to wait for these network guys to wake up and drink their morning coffee. Thanks again!
- jonah 13y agoIf the majority of the requests are coming from a certain set of IPs, configure your server to immediately drop those requests to reduce the load on your app.
- patio11 13y agorobots.txt is purely advisory and many of the people who code a crawler which could get in a state where it hits you hundreds of times a minute might not be great about actually respecting it. If it is a crawler, though, it is probably operating from a single C block, likely on AWS or what have you. You'd probably just drop them at the firewall. Simple, no fuss. I'm not a sysadmin, and I have nothing but respect for you and your team with the following recommendation: if it isn't obviously how to implement "drop that C block at the firewall" then find yourself a competent contract sysadmin, pay him for one hour of work today, then pay him for 10 hours of work over the next few weeks beefing up your infrastructure. It will be cheap at the price. If you're absolutely determined to do it yourself the magic word is "iptables" and may God have mercy on your soul, because iptables is the most obtuse piece of software I've ever had the misfortune of trying to configure. In the alternative, if you're actually facing a botnet, the expected behavior is lots of requests from widely distributed IPs. You'd be able to verify this with your access logs (which record IPs and timestamps) and a 10 or 20 line shell script to put a histogram of unique requests per IP. If it's a few requests from thousands of IPs per hour, then you have a bit of a challenge to address it on your end of things. My first totally-not-a-sysadmin advice would be to dial down dynamic content within the app for non-logged-in users and start aggressively caching pages, which might or might not be enough for you to weather the storm. Best of luck and skill.
- mdb31 13y ago50-100 requests per second doesn't sound like an attack (or at least not a very scary one, depending on request size...), more like a crawler or prefetcher gone haywire. Anyway, best way to figure out what's going on, is to generate a distinct list of IPs originating the bad traffic from your logs, and block those addresses at your firewall. If this doesn't stop the strange requests in 15 minutes or so, you may want to get some expert help. Your hosting provider may be able to be of some assistance here, but that greatly depends on their level of clue: many budget providers will just drop you at the first sign of denial-of-service related trouble, so do keep that in mind. Signing up for a specialized service like Cloudflare is probably your best bet, and should be pretty affordable as well (their web site lists their business plan with "advanced denial of service attack mitigation" as $200/month, and they suggest that if you call them they can have you protected pretty much immediately).
- panabee 13y agothanks for the suggestion! we're using singlehop ... definitely not a budget hosting provider. these requests have been ongoing since yesterday morning (~36 hours). we looked at cloudflare earlier today. do you have experience with them? does the solution really work?
- mdb31 13y agoNo personal experience with deploying Cloudflare, but they're pretty widely used (including by the very site you're on right now), so if you're really having urgent issues, I'd definitely give them a call.
- mbijon 13y agoCloudflare's plans that include DDoS protection are as good or better than typical hosting resellers. May not know enough about your site to make the decisions a good datacenter team could though. On the up-side Cloudflare really is there 24/7. Unfortunately the advanced DDoS support is only in their upper $$ plans. Might be worth it to keep your site revenue going though.
- taylorhou 13y agocloudflare.com? - I believe you could get everything swapped over to their DNS and start benefiting from their ability to ward off attackers within a couple of hours.
- good_guy 13y agoCan you tell us the site URL so we can take a look ?
- skimmas 13y agoshutdown servers. go outside. frustrate attackers by don't caring.
- deleted 13y ago[deleted]
- jbaudanza 13y agoWhat does your stack look like? If you're running a ruby app, check out rack-attack: https://github.com/kickstarter/rack-attack https://github.com/kickstarter/rack-attack It was created by kickstarter to fend off these kind of attacks.
- issa 13y agoMy only advice is to hire a consultant who can give you some advice. If your site is getting taken out by 50-100 requests per second (that is a very low number), you have some basic structural flaws. A few hours of time from a developer experienced with scaling will save you a lot of grief. Specifically, it sounds like you need a caching strategy (probably for all layers of your app, including the database). Super-general overview: Use memcached in front of your DB. Use a reverse proxy like Varnish and/or a CDN in front of your entire site. I don't want to make the obvious guess that this is a PHP site, but if it is, make sure you are using apc or xcache as well.
- caleb23 13y agoTo answer your first question, like people said already, pull a log from your servers with the IP's. If it is a lot of different IP addresses then that is a good sign of a DDOS attack of some kind. If it is a lot of requests from the same IP address or only a few IP addresses it probably is a search engine or something like which is causing the problem. You can also look up the IP addresses by simply Googling them to see who owns the IP. To fight off the attack immediately I would recommend switching your DNS over to CloudFlare's Pro or Business plan (that depends a lot though on your site's current configuration, the size of your budget, etc.) with certain settings, then configuring Sucuri CloudProxy with CloudFlare, and lastly implementing some additional security for your server on Single Hop. In terms of preventing future ones, you basically want to be proactive as possible in terms of preventing them, but that won't stop everything. The configuration I recommended above should be pretty good for most sites, but you might want to consider other DDOS mitigation companies as well and you could need something completely different depending on your business etc. Then you want to have a plan and several other things in place in case it was to get past the infrastructure you implemented to prevent it, so you can minimize the downtime you have etc. It is hard to tell whether that is a possible issue just based on the information you gave in terms of the sleeping MySQL connections. Lastly, depending on what the issue exactly is, it probably would be a good idea to make your site as static as possible, but that is difficult to do since your site is down right now. Once you get the site back up though and stable, you might consider doing this for a little bit. Lastly, having a separate site that you can deploy in the event of an attack is a good idea as well. If you want more help, shoot me an email at caleb.lane4@gmail.com. I hope that helped and you get this solved quickly.
- Udo 13y ago> 1) How do we confirm if the site is under attack? It could be some kind of attack, but it's so weak it barely qualifies. 50-100 requests is so low it's probably a search engine spider or a log-spamming bot. If request rates like these bring your site down, you have a serious software problem. You really should be able to handle at least a few hundred requests per second, especially if they're bogus requests going to non-existent URLs. > 2) If the site is under attack, how can we ward off the attack and prevent future ones? If this was a real attack, you'd be dealing with thousands of requests per second and a real spike in data volume. As a basic protection against things like these, get behind a reverse proxy right now. Cloudflare offers a good free plan, but it doesn't include SSL. I've had a good experience with X4B as a DDoS-protection reverse proxy recently. Their plans start at a few bucks per month, check them out. > 3) We seem to have a ton of sleeping MySQL connections that continue to crop up even after we try killing them. Is this symptomatic of a botnet attack? This question can't be meaningfully answered without knowing more about your setup. Generally, sleeping connections are fine - most web servers create a persistent connection pool. What's "a ton" in this case? I think the only really valuable advice to give here is: hire someone to review your setup, your software, to make some changes, and above all to give you a meaningful briefing. Contacting your provider isn't going to help.
- fragsworth 13y ago> If request rates like these bring your site down, you have a serious software problem. This true only if the requests that are being sent to them aren't malicious. A competent attacker without a botnet can try to find weak points in your application (clearing caches, writing data) and hit those places with just a few network connections.
- Udo 13y ago> A competent attacker without a botnet can try to find weak points in your application They can, but it this case it sounds unlikely. It's not feasible for a potential attacker to bother with this kind of analysis and then just send a few dozen requests per second. Instead they can just pay $5 to a Russian bot net operator who will inundate the site with hundreds of thousands of requests. Even from a single mid-range DSL line alone one could launch thousands of HTTP requests without breaking a sweat.
- cheald 13y agoI'm surprised I haven't heard it mentioned yet, but Varnish can ward off an awful lot of request floods. If these pages aren't changing frequently (particularly if they are outdated or shouldn't be accessible), then Varnish should be able to cache and respond with them trivially. This will take a tremendous amount of load off of your actual app servers. Additionally, Varnish has ACLs which make it possible to reject requests based on whatever patterns you choose (source IP, request structure, etc). While it puts you into an arms race, if you can identify a pattern in the bad traffic, you can short-circuit it before it ever becomes an actual problem. Black-holing poorly-behaved clients is very easy, and can let you absorb very high request concurrency with extremely little effort. Varnish also makes it very easy to locate URLs and IPs being used in these sorts of things, via varnishtop and varnishlog. For example, `varnishlog -i RxURL` gets you a constantly-updated list of the URLs being hit most frequently, or `varnishlog -i SessionOpen` will get you a list of the most common client IPs. With that kind of information, doing reactive filtering becomes very easy.
- User8712 13y agoYou could always try to improve the performance of your server and database configuration to handle 50-100 requests per second, and render the attack useless. A. Check the number of connections per IP. If any have an unusually high number, block them. B. Why is your site down? Database can't keep up? If so, why not, is it something simple like indexes not setup correctly? C. Did you actually check the load on your server? Double check it's actually being overloaded, and you don't have a configuration issue with server connections set too low, while your hardware is practically idling. D. How many pages are being accessed by the attack? Would you be able to cache them? E. Check server logs, and see if you can identify anything that separates these requests from your typical traffic.
- thaumaturgy 13y agoI took a quick look at the IP data you just released. Extracted IPs then unique-sorted them and took a few random samples. Everything was Eastern Europe, Russia, or Nigeria. There are some broadband IPs and some pppoe (!) IPs in there too, which might explain why it's not a very heavy attack. There are 500+ IPs listed. I've permabanned a few million IPs on my servers, and I bet there's some overlap. Gimme a sec to see if I can put together a quick script for you.
- gojomo 13y agoAs others have noted, it may not even be a malicious, targeted attack - just the side effect of other spamming or careless crawling/scraping. Reviewing your full logs, and comparing current traffic/behavior with 'normal' periods, may help further characterize what's happening. There's a presentation called "Running At 99 Percent: Surviving an Application DoS" by Ryan Huber that may be helpful. (Slides and video from a number of conference presentations can be found via a search on that title.) It highlights some of the ways even a trickle of the wrong kind of app-specific traffic can be enough to bottleneck weak points in your app... and talks about mitigation ideas, both as a matter of good design and emergency request-filtering via server-side tools.
- enneff 13y ago100 requests per second does not sound like a botnet attack. If it were you probably wouldn't be able to access your servers on their public interfaces for all the traffic.
- tluyben2 13y agoWhile running one of the largest free hosting providers we had many of these smaller attacks; usually some netstat & iptables incantations will ward them off. Something like netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n will show how many connections per IP for instance; with smaller attacks, usually, to actually make a dent in your webserver, the attackers will open many connections per IP. That helps you identify the 'bad' IPs. As someone else said; a lot of them are from eastern EU/Russia; you can use that information to be a bit more sure about what makes a bad IP a bad IP before you block it. Also check the Agent strings; often they all have the same Agent which is another way to filter. We used to have tons of scripts running to try to identify the 'enemies'; in the end the simplest thing was the best; block everything with a lot of connections and open them up 24 hours later.
- makmanalp 13y agoI wonder if this could be related to someone trying to mass-scrape you to get to your data cheap ... So instead of having to do tons of domain queries themselves, instead they use you to make the queries and just scrape the results into a large database. At least a 100 to 1 savings.
- SchizoDuckie 13y agoHave you checked your webserver logs for funny looking http requests? Have you considered that you're being SQLMapped by someone using the Tor Network? ( http://sqlmap.org http://sqlmap.org )