6 ms·
Mitigating DoS Attacks with Nginx
- DanBlake 11y agoFor all but the most basic attack, you really want a script putting these IP's into iptables. Using the application itself to block them still requires the connection setup/teardown resources to be used, as well as the application itself.
- giancarlostoro 11y agoI suppose that's where Lua would come in handy?
- iMerNibor 11y agoI personally have a access_by_lua script that counts accesses per ip and applies a (very generous) rate limit If the limit is reached the user will just be presented with a page explaining you hit a rate limit and a button that runs some javascript to verify you're not a bot which in turn whitelists the user This strategy has worked really well so far - havent been a target of too bad things yet though. Its a very good and cheap way to go for smaller sites though
- jsmeaton 11y agoCan a bot dedicated to your site just ping back whatever the javascript would have done anyway to cancel the rate limit?
- iMerNibor 11y agoIt /could/, the next step would be a captcha or something harder for bots to solve - haven't had to go that far yet though. But I usually only have to deal with script kiddies who rent out a botnet, enter a url and click the "attack" button
- hyperdunc 11y agoI can see how doing it lower level would be more efficient. Are there any scripts anyone could recommend as a starting point?
- SwellJoe 11y agofail2ban can watch the nginx logs for throttling and/or blocking messages and add iptables rules for you. I haven't read this all the way through, but on a cursory glance it looks reasonable: https://easyengine.io/tutorials/nginx/fail2ban/ https://easyengine.io/tutorials/nginx/fail2ban/
- XorNot 11y agoUrgh logwatching actively pains me these days. So much waste string parsing what was originally binary data anyway. I'm starting to think that we need some agreement where instead of logs, we just get apps to emit a stream of protocol buffers and a format string for the messages and data. Which does make me wonder if you couldn't LD_PRELOAD something which replaced fprintf and the like...
- SwellJoe 11y ago"So much waste string parsing what was originally binary data anyway." "So much" is pretty imprecise. How much waste do you believe string parsing incurs in this case?
- superuser2 11y agoStreams of plain text are what UNIX was built on. If you want binary APIs, look outside the *nix family.
- Ao7bei3s 11y agoOr modernize the applications. Throw out the ad-hoc formats and parsers, replace them with machine-readable equivalents. For example, systemd finally provides a logging system that allows structured logging with key/value fields.
- mgo 11y agoIf someone wants to take you down they'll just bombard you with traffic, and this won't help you there. Having been the victim of several DDoS attacks over the years, almost all of them haven't been on the application layer.
- aianus 11y agoCloudflare, for example, is good at preventing non-application-layer DDOS attacks but for application-layer they can't help much. This blog post is a good starting point for the kinds of strategies you need to fill that gap in protection.
- rodionos 11y agoWe tried it, setup was easy, but our response time for dynamic content increased by 150 millis so it didn't work for us. It's worth noting that their model is different from CDN - they proxy all of your traffic through their own servers.
- sciurus 11y agoThat's not atypical for a CDS these days; fastly and cloudfront can work the same way, e.g. https://aws.amazon.com/cloudfront/dynamic-content/ https://aws.amazon.com/cloudfront/dynamic-content/. How else do you expect them to cache and serve your dynamic content?
- dorfsmay 11y agoI don't recomend it, but you could use different domains for static vs dynamic.
- laumars 11y agoSome organisation do just that. But having your entire site behind CDN does have additional benefits besides mitigating DDoS attacks. Such as allowing you to handle other kinds of service outages more effectively (eg busy pages). They can offer you analytics, allow you to separate different traffic under the same domain name (sometimes handy for SEO), etc. Some CDN providers also do some cool stuff like enable IPv6 on your site even if your origin servers are only running IPv4 - but that's more a niche time saving feature than some "must have" deal breaker.
- Lanari 11y agoThis is really helpful and practical, when an app start getting more popular and people start writing scrapers and things like that sometimes they can mistakenly send an insane amount of requests, so this surely help on this cases since it make more since to solve that on the level of the server not the app itself. About DDoS I don't think there's any cheap solution for that...
- nodesocket 11y agoNginx is great, and I absolutely love it. However, if you're under a true DDoS attack, the box is going to be completely bogged down at the kernel level way before traffic is even close to being accepted and processed by NGINX. So this post is not very useful against a decent magnitude attack.
- nodesocket 11y agoHere is a sweet trick for dropping traffic with NGINX. When I mean drop, I mean, don't send a response, literally terminate the connection. location = / { if ($http_user_agent ~* foo|bar) { # return non-standard (NGINX only) 444 code # closes the connection without sending a response header return 444; } }
- NetStrikeForce 11y agoIf I'm not mistaken, the above implies accepting a TCP connection (3-way handshake) and a request from the attacker; then you look up the contents of the user-agent header and decide to stop replying based on its contents. This will get you nothing in a typical DDoS scenario, but thanks for sharing as it may come handy for other situations.
- creshal 11y agoFiltering like that also has a rather big performance impact on the nginx side and make things worse under moderate, but not DDoS-y, traffic conditions.
- kabdib 11y agoI'd love to be able to "drop" a connection without sending a FIN to the attacker. Leave them hanging on a timeout.
- mmaunder 11y agoI love nginx. But most of these are mitigation for DoS only targeting the web server, not DDoS. They're explaining how to throttle a single IP. The whole point of DDoS is to distribute the attack to bypass mechanisms that throttle single IP's (plus to amplify). Also the DDoS attacks that we've been hit with actually target our uplinks by saturating them with traffic, not our services. We have a 1 Gbps port and the last DDoS we were hit with was over 20 Gbps, which is a relatively small one. The mitigation we used was to have our hosting facility get their upstream provider to route the traffic through a layer 7 DDoS mitigation filter provided by an external company. It worked wonderfully. These are cool features, but when your link is saturated it doesn't matter what a daemon listening on a port does.
- codinghorror 11y agoAnd once IPV6 gets up to steam, welcome to a world of people with millions of "addresses" to attack from. One advantage of IPV4 is that it was accidentally pretty granular. That'll be a while, of course, but already we see attackers with access to a tremendous number of unique IP addresses in the IPV4 space.. they'll have many orders of magnitude more soon.
- dspillett 11y ago> welcome to a world of people with millions of "addresses" ... in the same /64 range for the most part, so as easy to block/filter/limit as one IPv4 address. You risk inconveniencing people who are assigned just a few addresses because you potentially end up blocking many of them due to the actions of a few on the same subnet, but you can't be held responsible for hosts/ISPs doing IPv6 wrong.
- Ao7bei3s 11y ago> ... in the same /64 range for the most part, so as easy to block/filter/limit as one IPv4 address. Possibly even easier, because of IPv4 deaggregation. (Because of IPv4 address scarcity, many providers have discontinous IPv4 address space. This is mostly a problem for the core, because it leads to much larger BGP routing tables.)
- ianamartin 11y agoI really wish Nginx didn't think it was so important to their business model to hold hostage the health check. That's a dealbreaker for me at their prices.
- dlecorfec 11y agoTake a look at Tengine, from Taobao. It's based on nginx.
- nikolay 11y agoIt is, just like OpenResty [0], but it lags severely behind recently - not sure why. Edit: Linked to Tengine in [1]. [0]: http://openresty.org/ http://openresty.org/ [1]: http://tengine.taobao.org/ http://tengine.taobao.org/
- druiid 11y agoThe Tengine module is available for compiling into the core nginx codebase. You just have to roll your own binary though https://github.com/yaoweibin/nginx_upstream_check_module https://github.com/yaoweibin/nginx_upstream_check_module
- nikolay 11y agoNginx, Varnish, etc. set criminal precedents by adding basic features to their paid offering. From cache purging to basic health checks - this arm-twisting ruins the experience! Instead of providing support and provide nice dashboards for paid users, they put the basics behind a paywall. I'm not sure how much they make from their Plus offerings, but I bet it's less than what Elastic collects in a much clever way!
- joosters 11y agoYou can't solve a traditional DDOS attack at the destination. No matter what you do with the incoming flood, the fact is, your bandwidth is full of the attack requests, leaving no room for legitimate traffic. Do what you like to the attack requests, but the pipe is still full. To mitigate a DDOS, you need to go upstream to your network providers and filter out the traffic before it reaches you.
- deleted 11y ago[deleted]
- icebraining 11y agoTrue, but not all DDoS involve bandwidth saturation, since if the site has a decent pipe, those are harder to achieve. Resource depletion based on forcing the server to perform heavy tasks is common.
- awqrre 11y agoAll the characteristics of DDoS attacks listed seem trivial to defeat... e.g.: Constantly spoof IP addresses, etc..