8 ms·
Common Nginx misconfigurations that leave your web server open to attack
- dddw 6y agoNice overview, thanks!
- chmod775 6y agoA lot of these look like not-so-great design choices in the way nginx is configured and how it handles paths. Sometimes the behavior that leads to security problems here may be desirable, but it probably shouldn't be the default. For instance "location /api {" probably shouldn't match "/api../" by default. Instead it should be treated like a file system would. The "prefix" matching should be a different configuration option like "prefix /api {".
- bawolff 6y agoIf someone does /api/../whatever, does nginx normalize that away automatically? Otherwise it seems like you could just do the attack directly (yes most clients wont let you make such a request, but that is easy to work around)
- bungle 6y agoIt does, but Nginx normalization is a bit dummy. It doesn't take into account the reserved chars for example.
- c0l0 6y agoIt's one of the shortcomings that I've come to loathe in nginx's declarative configuration language (and also other software products - Apache httpd is just as guilty). Everything just looks so innocent - but the devil is often in the details. So much implied nuance that you have to keep in mind when reading and especially when writing it. Sure it's expressive and also convenient (the latter at least as long as your configuration stays relatively simple), but something like varnish's imperative VCL that offers very little built-in magic sure is easier to reason about. I have come to consider that a feature.
- mhitza 6y agoWouldn't it be nice if all these programs started using at some point a proper embedded language for configuration? Lua, Tcl, Guile. I'd take any of those instead of the ad-hoc, declarative - but no really - languages that Apache and nginx use.
- feanaro 6y agoThat just seems like an even greater nightmare to me. Soon you would have to learn to read and understand a custom program in a Turing-complete language for each and every installation. The proper solution is a DSL, just a better DSl. Or perhaps a DSL embedded in something like dhall <https://dhall-lang.org/ https://dhall-lang.org/>, but definitely not a general-purpose programming language.
- mhitza 6y agoI was gonna consider including Dhall in my list, but then again nginx configuration has support for if statements. And from my past experience with HCL, sometimes a proper embedded programming language is better than whatever crazy DSL some developers can envision (see for loops in HCL)
- kevin_thibedeau 6y agoTcl is meant to be embedded for scripting and it can be made non-Turing complete by stripping away all but the bare minimum of commands. There is a built in mechanism for sandboxing untrusted input. If can easily be a safe, bullet proof configuration language.
- PowerBar 6y agoWhether JavaScript qualifies as a "proper embedded language" is up for debate, but nginx does support it for dynamic request handling. https://www.nginx.com/blog/introduction-nginscript/ https://www.nginx.com/blog/introduction-nginscript/
- mhitza 6y ago
- iforgotpassword 6y agoAgree. Nginx is probably a thousand times more powerful than lighttpd, but while setting up php with fastcgi on the latter was straight forward and easy to understand, with nginx you need a convoluted mess of includes, location directives, setting variables, then handling 404 is broken, there are ten different tutorials on how to do this and nine of them are wrong or open security holes and then you feel like a complete idiot.
- amenod 6y agoAgree. And there are other questionable design choices in this project too. by pet-peeve is an omission of `.htaccess`-like mechanism. There is even a page they have dedicated to this [0], where instead of looking at their users' problems and finding a suitable solution (like, only loading `.htaccess` every minute or when it changes), they argue that users don't actually have a use-case where they want to allow some limited configuration to 3rd parties. Someone even wrote a plugin that fixes that [1], but it is annoying (to say the least) that this is an option nginx developers say is "not needed" and "shouldn't be used". [0] https://www.nginx.com/resources/wiki/start/topics/examples/likeapache-htaccess/ https://www.nginx.com/resources/wiki/start/topics/examples/l... [1] https://github.com/e404/htaccess-for-nginx https://github.com/e404/htaccess-for-nginx
- latch 6y agoI realize that, at best, this is only tangentially related to security, but nginx's logging is quite frustrating. It'll log something that's completely out of your control (like invalid SSL requests) as a [crit] You end up having patterns in your log ingestion to drop errors. Or, and this is the security concern, you start to ignore nginx errors.
- cpach 6y agoYeah that’s a weird design choice for an application that is usually configured to allow requests from just about anyone on the Internet.
- avian 6y agoThis reminds me of a similar issue in Apache: it will log 500 “internal server error” when (if I recall correctly) clients close connection before SSL handshake is complete. Quite frustrating to try to figure out where your application is crashing only to find out there’s no bug and it’s only someone running a port scan or something.
- weinzierl 6y agoA common pit I fell into is the "inheritance" of add_header. If you use a single add_header at a lower level all add_headers at the higher levels are ignored for that server or location. As some headers have security implications this is an easy way to shoot yourself in the foot. Another security related point is the suppression of the server version. While nginx can omit the version number out-of-the-box, you unfortunately need an extension to remove the header completely.
- sneak 6y agoThat (the add_header thing) seems like a bug that should be fixed.
- megous 6y agoIt's intentional, and probably common to all "list" manipulating directives. It's the same behavior you get with fastcgi_param and similar, for example.
- HajiraSifre 6y agoYou just have to use ngx_headers_more instead of the built-in headers module. That add_header would get fixed (as a sibling comment states it should) is unlikely as it is intended to work that way: > There could be several add_header directives. These directives are inherited from the previous configuration level if and only if there are no add_header directives defined on the current level. It really is too bad that the functionality provided by ngx_headers_more isn't available out of the box, since it makes it a pain to use nginx on distributions that don't package it.
- weinzierl 6y agoIn addition to that, the ngx_headers_more module also solves the second issue I mentioned. It can be used to remove the Server header completely.
- 2ion 6y agoIt should also be noted that setting the server token as part of nginx core is part of their commercial offering [1]. [1] http://nginx.org/en/docs/http/ngx_http_core_module.html#server_tokens http://nginx.org/en/docs/http/ngx_http_core_module.html#serv...
- bennofs 6y agoThough deprecated, https://github.com/yandex/gixy https://github.com/yandex/gixy is a nice checker for these kind of issues.
- grawlinson 6y agoWhereabouts does it say that it's deprecated? The GitHub link doesn't state anything. Do you know of any alternatives? EDIT: Nevermind, I skipped the picture-stamp thingies.
- sharestuff 6y agoYou might find this follow up research interesting. It says gixy doesn't cover all: https://labs.detectify.com/2021/02/18/middleware-middleware-everywhere-and-lots-of-misconfigurations-to-fix/ https://labs.detectify.com/2021/02/18/middleware-middleware-...
- jzer0cool 6y agoAn example of the ../ bugs still be exploitable today.
- deleted 6y ago[deleted]
- holri 6y agoIs there something similar for Apache?
- Kinrany 6y agoDo Caddy/Envoy have similar issues? Bad UX is one of the reasons I still haven't learned to configure Nginx :(
- polyrand 6y agoI would recommend giving Caddy[0] a try. Most servers/reverse proxies need 10s of options to work more or less well. With Caddy, "correct" is the default, including having the best SSL management system (so you don't even need certbot) I've seen, and using HTTPS by default. It's true that it has some things missing (rate-limitng and weighted load balancing to name a few) that you can do in Nginx/Traefik/etc, but it's 100% worth it. Caddy also has a great extension system, so those things could easily be created as extensions. [0] https://caddyserver.com/ https://caddyserver.com/
- bottled_poe 6y agoI downvoted you because changing technology doesn’t inherently solve the concern of security. The other things your mentioned as strengths seem relatively equivalent to other web servers.
- shawabawa3 6y agoUsing Caddy _does_ solve the problem of "Common Nginx misconfigurations that leave your web server open to attack". Currently Caddy's defaults are secure and you don't need to worry about fiddling with settings to keep it that way
- Filligree 6y agoIt also has a lot fewer memory safety CVEs.
- efrecon 6y agoCaddy has a rate limiting plugin. Using it requires building a new Docker image, if necessary. https://github.com/hundertzehn/caddy-ratelimit https://github.com/hundertzehn/caddy-ratelimit
- jakearmitage 6y agoHow slow is it compared to nginx?
- megous 6y agoDefault for root is 'html' according to the docs (which is a relative path wrt /usr/share/nginx), not /etc/nginx. http://nginx.org/en/docs/http/ngx_http_core_module.html#root http://nginx.org/en/docs/http/ngx_http_core_module.html#root I guess it may depend on how nginx was configured during build. But for example on Debian this is not an issue.
- tylermenezes 6y agoI was confused about this one because their own study did not show a single person with /etc/nginx, and only a small handful with any vulnerable path.
- mjw_byrne 6y agoThis is a good example of the trade off between pretty/terse/clever and safe/correct/maintainable. Nginx is well-respected mature software but it's hard to see these issues as anything other than a design blunder. The trailing-slash path traversal thing looks to me like a file system analogue of SQL injection. I think d3.js is another example of this. It's obviously written with incredible skill but I could never get on with the ultra declarative and implicit style, it always felt like a fight. These days there seems to be a trend towards a verbose, explicit style, e.g. Zig (no hidden control flow - compare to C++'s operator overload-fest) and Go.
- secondcoming 6y agoWell there is a benefit in being able to read code and know exactly what's going on. For me, reading words requires less mental effort that mentally parsing glyphs.
- im3w1l 6y agoEarly mathematicians used words instead of symbols > To determine two quantities from their difference and product, multiply the product by four, then add the square of the difference and take the square root. Write this result down in two slots. Increase the first slot by the difference and decrease the second by the difference. Cut each slot in half to obtain the values of the two quantities.
- colejohnson66 6y agoIf I’m reading this right, this is saying, given: diff := abs(a - b) prod := a * b` Fine `a` and `b` by doing: temp := sqrt(prod * 4 + diff^2) a := (temp + diff) / 2 b := (temp - diff) / 2 That was a lot of words for something that (I feel) is easily expressed with symbols. It took me a minutes or two to figure out what you meant by “slot”
- jmt_ 6y agoWhen it comes to math at least, I think the key is finding the balance between symbolic and lexical representation. Some ideas are far to "wordy" to not use symbols. Some ideas have far to much depth to just explain with symbols. But using words to explain symbols? Very good in my experience during my math degree.
- deleted 6y ago[deleted]
- egberts 6y agoThis confabulated configuration syntax is why I discontinued using NGINX. This selection of NGINX came after a frustrated debugging session of Apache .htaccess as well. furthermore, unlike Apache specific IP port assignment capability, I once had to jerry-rig a dynamic configuration to tie NGINX to just one dynamic IP port out of many. Sorry, I’ve gone lighttpd and haven’t looked back since.
- tenebrisalietum 6y agoWell if you hate config syntax you can use rwasa which has ... absolutely none other than command line parameters.
- jlokier 6y agoThe one about proxy_pass blindly forwarding syntactically malformed requests and silently failing to process the response is astonishing. It doesn't appear to be documented. Looking through NginX documentation at http://nginx.org/en/docs/http/ngx_http_proxy_module.html http://nginx.org/en/docs/http/ngx_http_proxy_module.html I don't see anything (e.g. under proxy_hide_header) to say it's sometimes not applied, and there doesn't appear to be any option to prevent this blind forwarding. I would never have expected the backend to receive invalid HTTP from NginX, but more importantly it's not uncommon for backends to send an extra header or two to tell NginX how to serve the response, with NginX removing those headers before serving. How do you even handle this properly? Checking for valid HTTP might not be enough, as you need to exactly match whatever NginX's idea of valid is, rather than matching the HTTP spec.
- dheera 6y agoFrom TFA: XTTP/1.1 500 Error Content-Type: text/html Secret-Header: secret-info Secret info, should not be visible! What the hell backend would respond with this? I just tested this with a simple hello world NodeJS backend: const express = require('express'); const app = express(); app.get('/', (req, res) => { res.send('Hello World!') }); app.listen(3000, () => { console.log('Listening'); }); And then tried the example: $ telnet localhost 3000 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. GET /? XTTP/1.1 Host: 127.0.0.1 Connection: closeHTTP/1.1 400 Bad Request Connection: close Connection closed by foreign host. Seems like the backend responded with a valid HTTP response even though my request was invalid. Of course that doesn't speak for all backend frameworks people would use, but it would occur to me that a well-designed backend would always speak proper HTTP even if the input isn't proper HTTP, and if the backend receives a bad HTTP request it should immediately send back a 400 (not 500) and never pass it on to the business logic. My brief test above seems to suggest that Express does indeed behave this way. Separately, configure your backends to not spit out secret info in production mode and you should not have to actually worry about this.
- throwaway894345 6y ago
- neals 6y agoI've been running Nginx for more then 10 years now... Is there anything "new"? I know about serverless, but anything else that makes the webserver part easier and safe?
- sharestuff 6y agoThere was a follow-up to this released last week by Frans Rosen on Detectify Labs that looked into middleware in general and still applicable to nginx: https://labs.detectify.com/2021/02/18/middleware-middleware-everywhere-and-lots-of-misconfigurations-to-fix/ https://labs.detectify.com/2021/02/18/middleware-middleware-...