12 ms·

Avoiding the top Nginx configuration mistakes

417 points by anotherevan 5y ago

michelb 5y ago
As an Apache user for relatively simple websites (wordpress, laravel, not high-load web-apps), under moderate load, should I ever migrate to NGINX? I love how easy Apache is to configure. I'm using the event module with PHP-FPM, would I even notice a performance improvement?
- porker 5y ago
  Slowloris attacks are the only reason I've switched from Apache to nginx. AFAIK Apache using event/worker still has no protection.
- shantnutiwari 5y ago
  Sometime ago there was a big push "Apache is old fashioned and clunky! Nginx is sooooo powerful! Move to Nginx!!" Now we are seeing the opposite backlash. As someone who doesnt know much about this topic, I also wish there was an easy answer :/
- sgjohnson 5y ago
  It’s always worth learning NGINX, as it is a truly powerful tools, but it’s often PITA to configure, and there are a lot of pitfalls if you use virtualhosts.
  - gjvc 5y ago
    Linux distributions of nginx sometimes have their conf.d/ and available-sites.d/ directories in /etc/nginx/ which sometimes can confuse matters. With software packages such as nginx (in general, not in the context of Linux distributions mentioned above), it's sometimes easier / clearer to start with a completely empty configuration file, and add just enough to make it work (TDD-style), and / or stop it complaining. If you can explain / defend / justify the presence of every line, you stand a much better chance of knowing how it will behave.
- mekster 5y ago
  Perhaps some benchmark would convince one.
- thejohnconway 5y ago
  I'm just a hobbyist, so take this as you will, but my experience with NGINX is bad. Sure it might be fast (does that even matter at normal scales?), but it configuration is a pain in the neck, as this article conveys. Common things seemed hard and fragile to me, like getting the reverse proxy set up with Let's Encrypt. As it's just a thin layer on what I am actually doing, spending days or weeks learning the proper way to do things felt like an imposition on my time. At some point I broke my config and gave up on it – I moved my whole site to a new VPS without a reverse proxy and encrypted through Cloudflare. I've since moved to Caddy, and uh, wow, this is what it's supposed to be like!
- NorwegianDude 5y ago
  Apache is fine. No reason to switch to nginx unless there is something nginx offers that apache doesn't. In earlier days a lot of people was under the impression that nginx was much faster for PHP, but that was not true. Apache with mod_php was faster, but the issue was memory consumption for non php files as the php module was loaded on every request, even if it wasn't php. When using php-fpm you don't need the php module on every request as php-fpm is contacted as needed. This does increase latency and decrease performance for php requests, but the difference is indistinguishable for most people. Slowloris attacks was/is one issue with non event-based webservers. Both nginx and apache can handle a six digit number of php requests each second on a low end server. You application(Wordpress, Laravel) will without a doubt be a bottleneck before the webserver becomes the bottleneck.
  - tyingq 5y ago
    Most of the benchmarks I'd seen showing Nginx as significantly faster than Apache boiled down to configuration. Having mod_rewrite turned on with AllowOverride for Apache, for example, slows it down quite a lot.
  - matja 5y ago
    I remember one of the early "selling points" of nginx was it didn't traverse the directory tree for a .htaccess file to dynamically load config from, for each request, when that was specifically a "selling point" of Apache, which is very useful for shared hosting contexts when users don't have access to change the webserver config.
    - NorwegianDude 5y ago
      Considering that that's something you can choose to do with apache isn't a selling point for nginx. It's an optional feature nginx lacks. But yeah, there was a lot of weird and misinformed reasons for why people recommended others to use nginx no matter what the use case was.
- 2000UltraDeluxe 5y ago
  If you need to proxy the requests to your Apache server(s), then sure, Nginx is really nice. It's got proper HTTP/2 support unlike Apache, and it also supports proxying mail and other connections if needed. There's little reason to replace Apache completely, though.
xPaw 5y ago
Another big mistake with defaults is that you need an explicit `server` block with `default_server` for all the `listen` directives you have. Otherwise if you have a listen directive for some host that doesn't have a default, this server will be used as the default for this particular listen directive. For example doing `listen 80` and then requesting any non matching host will still return this server.
KronisLV 5y ago
What upsets me about Nginx the most is that it takes the fail-fast approach, which isn't entirely suitable for combining it with a container workflow, at least in situations where you're running it without something like Kubernetes (which has its own customizations). For example, currently i run my own Nginx ingress with Docker Swarm: service A / B / C <--> Nginx <--> browser The reverse proxy configuration feels easier to do than that of Apache2/httpd, for example: server { listen 80; server_name app.test.my-company.com; return 301 https://$host$request_uri; } server { listen 443 ssl http2; server_name app.test.my-company.com; # Configuration for SSL/TLS certificates ssl_certificate /app/certs/test.my-company.com.crt; ssl_certificate_key /app/certs/test.my-company.com.key; # Proxy headers proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; # Reverse proxy location / { resolver 127.0.0.11 valid=30s; # Docker DNS proxy_pass http://my-app:8080/; proxy_redirect default; } } However, suppose that the "my-app" container has a health check set up for it. What does that mean? Docker Swarm won't route any traffic to it until it's finished initialization. Except that it also means that DNS requests won't return anything for "my-app" until at least one healthcheck passes. That in of itself shouldn't be too problematic, right? Any web server surely would just log that one request as an error and return an error until eventually everything would work. Wrong. Nginx does throw the error and the request fails, but the ENTIRE web server goes down: nginx: [emerg] host not found in upstream "my-app" What does that mean? If you have 1 Nginx instance on the server that needs to proxy 10 apps in total, one app not being available will prevent the other 9 from working! Whoever thought that such defaults would be okay has reasoning that is beyond me, especially considering that the Caddy web server does something similar and will go down if 1 out of 10 sites fails to provision the SSL/TLS certificate, instead of letting the others work - completely opposite of what someone who cares about resiliency would want! But fear not, supposedly there's a solution! # Reverse proxy location / { resolver 127.0.0.11 valid=30s; # Docker DNS set $proxy_server my-app; proxy_pass http://$proxy_server:8080/; proxy_redirect default; } Except that, you know, that also doesn't work: nginx: [emerg] "proxy_redirect default" cannot be used with "proxy_pass" directive with variables Now, that's not an issue in my case, since i just jotted down the "proxy_redirect default;" value to remember what it does, but it's unfortunate to hear that it is neither ignored (since it's the default value which is functionally identical to it not being there), nor could other values be used: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_redirect http://nginx.org/en/docs/http/ngx_http_proxy_module.html#pro... In summary, it's nice that there are workaround for problems like that, but it's a bit odd that the web server is otherwise so quick to fall over and die, as well as the fact that the defaults aren't oriented towards maximum reliability and resilience. That said, i still do like Nginx and actually wrote about how to use it as a reverse proxy on my blog a while back: https://blog.kronis.dev/tutorials/how-to-use-nginx-to-proxy-your-front-end-and-back-end https://blog.kronis.dev/tutorials/how-to-use-nginx-to-proxy-... But at the same time, in situations where load isn't high enough for the web server to be the bottleneck, it's also helpful to evaluate the alternatives, notably: Apache2/httpd, Caddy and Traefik. It was actually pretty recently that Apache2/httpd got mod_md, which allows provisioning Let's Encrypt certificates out of the box, which is certainly promising: https://httpd.apache.org/docs/trunk/mod/mod_md.html https://httpd.apache.org/docs/trunk/mod/mod_md.html
bazzargh 5y ago
One we see a lot is this: https://trac.nginx.org/nginx/ticket/1151 https://trac.nginx.org/nginx/ticket/1151 worker_processes auto; in a kubernetes environment where the nginx container only has 1 cpu share, but is running on a 64 core node...this does the wrong thing and launches dozens of workers, which do nothing but eat memory. According to the bug above they've not fixed it because they can't tell the difference between this case and other cases where your cpus may be restricted. While that's true, other systems don't just ignore the problem: http://mail.openjdk.java.net/pipermail/hotspot-dev/2019-January/036120.html http://mail.openjdk.java.net/pipermail/hotspot-dev/2019-Janu... (we also commonly see issues with buffering, and proxy host header not being set; the latter is an issue when code moves from an environment with physical hosts - which don't care about the host header - to virtual ones)
xorcist 5y ago
There are some issues specific to nginx, and probably not the most common errors made. It's fundamentally hard to manipulate http requests without being knowing all forms paths and headers can take. Careless use of regexps in http routing is a common source of problems. Things like uri encoding, parameters as subdirectories and control characters is easy to get wrong. Treating user supplied headers as trusted is also easy to get wrong. It's not uncommon to see configurations which are remotely exploitable, which is another level of bad from what is described in the article.
jurschreuder 5y ago
Editing the nginx config until worked correctly has cost me so many days of my life. Luckily there are very good Golang alternatives for 99% of use cases now.
alberth 5y ago
What is the current recommended web server these days? Is it still nginx? I know caddy gets a lot of attention but always assumed it was for smaller/less-trafficked sites (maybe I'm wrong).
- fraktl 5y ago
- mholt 5y ago
  Caddy is used by large enterprises and is processing hundreds of thousands of dollars of revenue per hour at one company I know of. You can definitely use it for high volume sites.
LinAGKar 5y ago
So add doesn't (always) add, and instead you have to repeat yourself all over the place? That's just stupid.
schwag09 5y ago
This article briefly mentions a very useful analysis tool for NGINX configuration: Gixy. It looks for the following misconfigurations[0]: - [ssrf] Server Side Request Forgery - [http_splitting] HTTP Splitting - [origins] Problems with referrer/origin validation - [add_header_redefinition] Redefining of response headers by "add_header" directive - [host_spoofing] Request's Host header forgery - [valid_referers] none in valid_referers - [add_header_multiline] Multiline response headers - [alias_traversal] Path traversal via misconfigured alias The alias traversal gotcha is one of the most pernicious I've seen. A single, seemingly innocuous '/' is the difference between a path traversal vulnerability or not. [0]: https://github.com/yandex/gixy#what-it-can-do https://github.com/yandex/gixy#what-it-can-do
bennettdams 5y ago
Isn't it ironic that even the writers of this article don't follow these rules in the examples on that page? There is "Mistake 3: Not Enabling Keepalive Connections to Upstream Servers", where it says: In the location{} block that forwards requests to an upstream group, include the following directives along with the proxy_pass directive: proxy_http_version 1.1; proxy_set_header "Connection" ""; Then, at "Mistake 10: Not Taking Advantage of Upstream Groups", the "location" block does not include these two directives, even though the "upstream" block contains a "keepalive" directive.
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
kseifried 5y ago
I've assigned this the Global Security Database (GSD) ID GSD-2022-1000285 (https://raw.globalsecuritydatabase.org/GSD-2022-1000285 https://raw.globalsecuritydatabase.org/GSD-2022-1000285) and started a discussion about these informational entries at https://groups.google.com/a/groups.cloudsecurityalliance.org/g/gsd/c/t3l9DJXmN3A https://groups.google.com/a/groups.cloudsecurityalliance.org... TL;DR: if you run nginx I assume you want to know about these types of potential problems, so sticking it into the security database makes a lot of sense in my opinion for discoverability.
jdsleppy 5y ago
Nginx config seems like a bottomless pit. It gives me the painful feeling that no matter what I have done with it, there's some security or performance issue still there. Does this mean I should just stick with a more managed solution? Or is this a common feeling that I should just learn to live with? For my fun projects I use Caddy with mostly default settings.
- freedomben 5y ago
  I wouldn't be that worried about it. Sure there are ways to optimize it more and squeeze every drop of juice out the grape, but most of the time you're gonna be just fine. It might be the difference between running an extra app server or not. Now that said nginx website has a ton of great examples. You can usually copy/paste them into whatever you need pretty easily, and once it's done it's not likely to change. I would especially do this if nginx is your internet-facing load balancer and has multiple upstreams behind it. Plenty of good examples out there.
giobox 5y ago
nginx config increasingly feels like a bunch of foot guns waiting to go off for me. For smaller projects, I've really enjoyed Caddy's "best practice by default" approach to webserver/loadbalancer config where you can generally only choose to remove best practices, not try to build them up from scratch. Has SSL and LetsEncrypt enabled by default, etc. This avoids a lot of the pitfalls of incorrect webserver config altogether, keeps the configuration file really small and easy to read. I wish this style of config was more common. > https://caddyserver.com https://caddyserver.com
- KronisLV 5y ago
  I actually really like Caddy, though some of its defaults at least historically have been odd, such as responding with 200 where other web servers would respond with a 502 or something similar: https://caddy.community/t/why-does-caddy-return-an-empty-200-response-for-a-non-configured-host/7685 https://caddy.community/t/why-does-caddy-return-an-empty-200... Also, this is a bit of a personal preference, but v1 Caddy felt like maybe a bit easier to get started with than v2 Caddy, though sadly was abandoned. I'm sure that there were good reasons for doing it, but the few forks that started out from it with the intent to maintain it never really went everywhere, so v2 is the only possibility nowadays, unless you want to maintain it yourself or like dead software: https://github.com/WedgeServer/wedge https://github.com/WedgeServer/wedge Of course, that's not a criticism of Caddy itself, just how we as an industry sometimes need rewrites and that makes us keep up with the churn.
  - francislavoie 5y ago
    The fact of the matter was that _servers aren't actually easy_. Caddy v1 tried to give the illusion that it is, but it just caused more problems, really. The complete rewrite for Caddy v2 was necessary to make Caddy flexible enough to solve like half or more of the open issues the Caddy repo had at that point. Flexibility and simplicity are often at odds. But Caddy tries to keep smart defaults in general. Also, any time someone brings up the malicious fork "wedge", it saddens me. That was done at literally the most stressful moment in the project's history, and it only made things worse for everyone. I really hope people learn to chill out and give maintainers a bit of time to breathe and respond instead of taking such hostile measures like that. (You can probably pretty easily dig up the Github issue with the discussion if you really care to see what went down. Sigh.)
cosmiccrisp 5y ago
I remember someone posting an article of common nginx config security mistakes but can't find it. Does anyone remember? I would like to read through that again as well.
- WaxedChewbacca 5y ago
- iancarroll 5y ago
  Not a comprehensive article, but the worst security footgun, by far (IMO), is $uri. It’s completely unsafe to use $uri in basically any directives! You cannot redirect using it, proxy_pass using it, or you will have a bad time. https://reversebrain.github.io/2021/03/29/The-story-of-Nginx-and-uri-variable/ https://reversebrain.github.io/2021/03/29/The-story-of-Nginx...
  - cosmiccrisp 5y ago
    Thanks! That is good info.
rr808 5y ago
Nginx would benefit from a simpler higher level instruction set or an official 2-5 recipes as I'm sure 99% of the implementations doing a few basic things and dont need the ability to screw it up.
- binarymax 5y ago
  Definitely. Been using it for years for many projects and it’s always the same needs: Let’s encrypt + static files + upstream reverse proxy. I had no idea about keepalive not being default - why isn’t this in all the examples and tutorials?
  - ReactiveJelly 5y ago
    For that I use linuxserver's SWAG container. Let's Encrypt automates cert renewal, and SWAG automates having to remember how in the heck to set up LE.
  - zhte415 5y ago
    I'm sure there' space in the tutorial area. A general point for $tech: If any reader is thinking of writing a tutorial, do check if it's been done before. Likely a grossly simplified example of how to do X while the person reading it has read upteen tutorials on how to do X, Y and Z but still doesn't know where to keep configuration files, or how anything interacts with anything else. Instead, think of writing Series Of Tutorials That Together Build Moderately Fully Fledged Thing Including Satellite Stuff Like, in this case, Nginx, or SMS, or why XX characters for an email address string, or whatever of the plethora of things to Make It Work are needed. Some books do this. Not a huge number though, certainly not compared to hit-and-run tutorials. Call them books. Call them Tutorial Super Series. I don't think the little more time invested would not generate outsized returns.
deleted 5y ago
[deleted]
The_rationalist 5y ago
cosmotic 5y ago
Nearly all of these are approximately "users did something they expected to work, but it didnt; shame on them" or "We chose a safe but mostly bad default, shame on users for not realizing it". I understand the rational for not changing old defaults in fear of breaking existing configs. However, maybe a release or two of deprecation or better optimized defaults under new names or a new (kinda gross but...) strict mode which is opt-in but is the new default for new instalations. 1. Warn the user on startup of potential misconfiguration 2. Allow error_log off to work, as users are expecting it to 3. Enable keep alive by default 4. Acknowledge the confusing syntax and find a better solution than blaming users 5. Warn the user on startup of impact of proxy_buffering off 6. See 4 7. See 1 8. See 1 9. Fix the proxy hash algorithm so that it hashes the whole address (why only first 3 octets?) or give it a new name; see also 1 10. Detect possible optimizations on startup and implement automatically; why ask the user to config something that can potentially be detected automatically?
- josephg 5y ago
  Absolutely. If you (nginx) know how your product should be configured, you should configure it that way out of the box. Don’t blame users. And if backward compatibility is important, take a leaf out of Rust’s book and make a nginx-config-version 2; directive or something. With this set, you can safely use the new defaults. Use it across all example configurations on the website and in distribution packages.
- klaussilveira 5y ago
  It is interesting to see nginx commit the same mistakes that led Apache to its demise.
- BeFlatXIII 5y ago
  > However, maybe a release or two of deprecation or better optimized defaults under new names or a new (kinda gross but...) strict mode which is opt-in but is the new default for new instalations. Something like this or a \version "2.12.0" command would be massively useful. The alternative is a sea of undisclosed footguns that are blindingly obvious once you know they exist but otherwise are well-camouflaged. Ok, the other alternative is for every future developer to brainlessly spam a long incantation of sensible values that they swear they'll learn about later just so a Fortune 500 early-adopter company can have smoother upgrades.
nemothekid 5y ago
I'm surprised the "set a resolver with a dns ttl" isn't on here; it's probably my only headache with nginx. If you have an upstream that you specify with a hostname, nginx will lookup the host, cache it, and never refresh it by default. I don't know why it happens to me so often, but I will set up nginx containers that proxy to some load balanced service (via DNS), and then one day the IPs will change and that service would break because I forgot to set resolver.
- amdelamar 5y ago
  This is my top issue with open-source nginx. Nginx plus has the solution though with its `resolver` directive. https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#resolve https://docs.nginx.com/nginx/admin-guide/load-balancer/http-...
  - megous 5y ago
    Opensource nginx has the same resolver directive.
    - kzemek 5y ago
      But servers in upstream block don't have a 'resolve' setting in non-plus versions. 'resolver' still does something in the open source version, but requires some configuration contortions.
- cyounkins 5y ago
  This issue has bitten me multiple times. You must set the resolver AND use the domain name in a variable. See here: https://www.nginx.com/blog/dns-service-discovery-nginx-plus/ https://www.nginx.com/blog/dns-service-discovery-nginx-plus/ under "Setting the Domain Name in a Variable". The other options ignore the record TTL or require Nginx Plus. More details here: https://github.com/DmitryFillo/nginx-proxy-pitfalls https://github.com/DmitryFillo/nginx-proxy-pitfalls
  - egberts1 5y ago
lokar 5y ago
Why would you pick nginx over envoy?
- mrweasel 5y ago
  They aren't really the same type of product. Envoy is mostly an alternative to something like HAProxy, or Træfik. You wouldn't pick Envoy over Nginx (or Apache) if you need a webserver, than wouldn't make sense. Envoy solves a very specific problem, Nginx solves a bunch of problems. Also if you're using Nginx everywhere else already, you might want to use a product you're comfortable with, even in the cases where Envoy could work for you. E.g. we use Apache almost everywhere, because performance is "good enough", and it allows us to use the same product for a lot of different client, due to the large feature set.
- manigandham 5y ago
  nginx is a reverse proxy and app server. It has all the major features to handle front/edge tasks (like TLS termination), serve static files, and run entire applications in-process given the correct modules. nginx has entire ecosystems around this with OpenResty and even runs the edge workload for Cloudflare's CDN and Algolia's search service. envoy is a L7 proxy focused on full duplex networking across multiple protocols with transparency and observability. It can also handle front/edge tasks although the configuration system is a magnitude more complex than nginx and it's also missing the app server abilities. If you use envoy already then it might be easier to deploy it to the edge as well, and it's often used that way in Kubernetes as both service-mesh and ingress - but if you just need typical reverse-proxy/app serving than nginx is still far simpler to setup and maintain.
- deleted 5y ago
  [deleted]
- jcdavis 5y ago
  Serves static content (which envoy still can't do AFAICT) Simpler to set up for a basic proxy. For more complex usage though, nginx is certainly outclassed (at least the OSS version)
- mike_d 5y ago
jrtc27 5y ago
> The if directive is tricky to use, especially in location{} blocks. It often doesn’t do what you expect and can even cause segfaults. That's not user error. That's lazy programming and a bug. If your code segfaults because of user input you weren't expecting, that's on you as a programmer.
- mdaniel 5y ago
  It wouldn't surprise me if they consider that WONTFIX given they wrote a whole wiki page entitled "If is Evil": https://www.nginx.com/resources/wiki/start/topics/depth/ifisevil/ https://www.nginx.com/resources/wiki/start/topics/depth/ifis...
  - jrtc27 5y ago
    Oh I know. Hardly inspires confidence; all of those examples are clear bugs declared features because of poor programming practices.
  - Dylan16807 5y ago
    I can understand why certain directives don't work inside an if, even if segfault is dumb. Why are the empty if blocks breaking things outside of them?
egberts1 5y ago
We still cannot bind NGINX to a specific interface. sad
- jesterson 5y ago
  What do you mean here? If you are referring to network interface, why there is a problem to bind to it?
  - sgjohnson 5y ago
    You can’t bind nginx to a specific network interface. You can bind it to a specific IP address though (and then, by extension, also to an interface, but you need to know the IP).
    - jesterson 5y ago
      That's right, although I am struggling to think of use cases where you may need to assign it to network interface with dynamic IP allocation (otherwise you'd know the IP beforehand).
      - sgjohnson 5y ago
        > although I am struggling to think of use cases where you may need to assign it to network interface with dynamic IP allocation Some failover configurations come to mind, in particular the one where the box doesn’t get an IP address on the said interface until the failover kicks in, but I don’t know why would anyone run this kind of a setup without NAT.
        egberts1 5y ago
        I did that bind with DHCP server failover.
      - egberts1 5y ago
        My use case is simple. I have five disparate IP subnets whose firewall must NOW have to ascertain that ONLY a certain subnet can access this unbinded netdev … in case the NGINX (or any daemon) starts to sniff unwarranted IP subnet traffic. And yes, you can bind to a dynamic IP assigned netdev. And yes , I’ve had to replace NGINX with a web server that can do a netdev bind.
oxplot 5y ago
Traefik has been my go-to reverse proxy for quite a while now. The main advantages is its docker/k8s and native ACME support. You can configure services, config, routes, etc. via docker labels. When running under compose, this makes it very clean and convenient to maintain config. It also has plenty of useful middlewares and has a prometheus exporter for its internal stats.
- mekster 5y ago
  It's good we have many options to choose from. Apache, nginx, caddy, Traefik, HAProxy, lighttpd, Varnish, Envoy are likely to be the major ones?
vishnugupta 5y ago
Of late (~1 year) I've migrated to AWS ALB from Nginx and recommend it if you are on AWS ecosystem. Being a Layer-7 load balancer, ALB is (almost) feature compatible with Nginx. You can do route based, header based load balancing, terminate SSL and so on. For an internet facing app my preferred setup now a days is [CloudFront → ALB → application].
winrid 5y ago
The big one for me was the SSL cache size config generated by certbot combined with nchan. That caused so many production issues w/ many concurrent connections.
ranger207 5y ago
My biggest gripe with nginx is the documentation. I ran into an issue with proxy_buffering being off yet I was still getting 502 errors because the upstream was passing too large headers for the default 4k buffers. I naively assumed that with proxy buffering off that it would send data to the client immediately and, y'know, not buffer it, as [0] suggests. Turns out that turning proxy_buffering off doesn't turn proxy buffering completely off. While working on that issue, I also needed to set the proxy buffer size. Quick, what's the difference between proxy_buffer_size [1] and the size argument of proxy_buffers [2]? I still don't know, because the docs for each directive sound to me like they're restating the same exact thing. [0] https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering https://nginx.org/en/docs/http/ngx_http_proxy_module.html#pr... [1] https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size https://nginx.org/en/docs/http/ngx_http_proxy_module.html#pr... [2] https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers https://nginx.org/en/docs/http/ngx_http_proxy_module.html#pr...
- Dylan16807 5y ago
  Looking at that documentation has me even more confused. The article says "Proxy buffering means that NGINX stores the response from a server in internal buffers as it comes in, and doesn’t start sending data to the client until the entire response is buffered." But those parameters say nothing about waiting to send data to the client, and in fact some of them like proxy_busy_buffers_size imply the exact opposite.
  - barrkel 5y ago
    Logically the purpose of proxy buffering is to prevent server death from clients holding connections open by reading the data very slowly (slowloris attack). But this doesn't require waiting for the entire upstream response before sending bytes to the client; merely that reading from upstream doesn't stop when the client is reading slowly. It looks like nginx does this while still preventing slowloris. The parameters mentioned look like they're about tuning the relative memory costs of sending down the first bit of data in a response, which may contain enough information on the client side to do something interesting. proxy_buffer_size looks like it's just the first buffer, while the size argument to proxy_buffers applies to all the buffers.
exciteabletom 5y ago
What are some alternatives to NGINX? Is there a maintained fork with a better config? I know of Apache, but that seems to be falling out of favour lately.
- jimmySixDOF 5y ago
  HAProxy is one and I'd like to see a breakdown of how these config issues either spillover or are dealt with differently in that environment.
  - exciteabletom 5y ago
    Thanks so much for replying! Just looked at some example configs and it looks way slicker than nginx.
  - mrweasel 5y ago
    There's an overlap in features between Nginx and HAProxy, but one isn't necessarily a replacement for the other. Nginx is a full webserver, HAProxy isn't a webserver.
- ahmetkun 5y ago
  > I know of Apache, but that seems to be falling out of favour lately. lately? more like for the last 10 years afaict.
- j_autumn 5y ago
  I like Caddy: https://caddyserver.com/ https://caddyserver.com/ Nice community and enough features for my (basic) use cases. The only thing that I don’t quite like is the logging format.
  - exciteabletom 5y ago
    The automatic certificate renewal is a killer feature. Goodbye dodgy cronjobs!
- mekster 5y ago

417 points by anotherevan 5y ago

As an Apache user for relatively simple websites (wordpress, laravel, not high-load web-apps), under moderate load, should I ever migrate to NGINX? I love how easy Apache is to configure. I'm using the event module with PHP-FPM, would I even notice a performance improvement?

porker 5y ago
Slowloris attacks are the only reason I've switched from Apache to nginx. AFAIK Apache using event/worker still has no protection.
shantnutiwari 5y ago
Sometime ago there was a big push "Apache is old fashioned and clunky! Nginx is sooooo powerful! Move to Nginx!!" Now we are seeing the opposite backlash. As someone who doesnt know much about this topic, I also wish there was an easy answer :/
sgjohnson 5y ago
It’s always worth learning NGINX, as it is a truly powerful tools, but it’s often PITA to configure, and there are a lot of pitfalls if you use virtualhosts.
- gjvc 5y ago
  Linux distributions of nginx sometimes have their conf.d/ and available-sites.d/ directories in /etc/nginx/ which sometimes can confuse matters. With software packages such as nginx (in general, not in the context of Linux distributions mentioned above), it's sometimes easier / clearer to start with a completely empty configuration file, and add just enough to make it work (TDD-style), and / or stop it complaining. If you can explain / defend / justify the presence of every line, you stand a much better chance of knowing how it will behave.
mekster 5y ago
Perhaps some benchmark would convince one.
thejohnconway 5y ago
I'm just a hobbyist, so take this as you will, but my experience with NGINX is bad. Sure it might be fast (does that even matter at normal scales?), but it configuration is a pain in the neck, as this article conveys. Common things seemed hard and fragile to me, like getting the reverse proxy set up with Let's Encrypt. As it's just a thin layer on what I am actually doing, spending days or weeks learning the proper way to do things felt like an imposition on my time. At some point I broke my config and gave up on it – I moved my whole site to a new VPS without a reverse proxy and encrypted through Cloudflare. I've since moved to Caddy, and uh, wow, this is what it's supposed to be like!
NorwegianDude 5y ago
Apache is fine. No reason to switch to nginx unless there is something nginx offers that apache doesn't. In earlier days a lot of people was under the impression that nginx was much faster for PHP, but that was not true. Apache with mod_php was faster, but the issue was memory consumption for non php files as the php module was loaded on every request, even if it wasn't php. When using php-fpm you don't need the php module on every request as php-fpm is contacted as needed. This does increase latency and decrease performance for php requests, but the difference is indistinguishable for most people. Slowloris attacks was/is one issue with non event-based webservers. Both nginx and apache can handle a six digit number of php requests each second on a low end server. You application(Wordpress, Laravel) will without a doubt be a bottleneck before the webserver becomes the bottleneck.
- tyingq 5y ago
  Most of the benchmarks I'd seen showing Nginx as significantly faster than Apache boiled down to configuration. Having mod_rewrite turned on with AllowOverride for Apache, for example, slows it down quite a lot.
- matja 5y ago
  I remember one of the early "selling points" of nginx was it didn't traverse the directory tree for a .htaccess file to dynamically load config from, for each request, when that was specifically a "selling point" of Apache, which is very useful for shared hosting contexts when users don't have access to change the webserver config.
  - NorwegianDude 5y ago
    Considering that that's something you can choose to do with apache isn't a selling point for nginx. It's an optional feature nginx lacks. But yeah, there was a lot of weird and misinformed reasons for why people recommended others to use nginx no matter what the use case was.
2000UltraDeluxe 5y ago
If you need to proxy the requests to your Apache server(s), then sure, Nginx is really nice. It's got proper HTTP/2 support unlike Apache, and it also supports proxying mail and other connections if needed. There's little reason to replace Apache completely, though.

xPaw 5y ago

Another big mistake with defaults is that you need an explicit `server` block with `default_server` for all the `listen` directives you have. Otherwise if you have a listen directive for some host that doesn't have a default, this server will be used as the default for this particular listen directive. For example doing `listen 80` and then requesting any non matching host will still return this server.

KronisLV 5y ago

What upsets me about Nginx the most is that it takes the fail-fast approach, which isn't entirely suitable for combining it with a container workflow, at least in situations where you're running it without something like Kubernetes (which has its own customizations). For example, currently i run my own Nginx ingress with Docker Swarm: service A / B / C <--> Nginx <--> browser The reverse proxy configuration feels easier to do than that of Apache2/httpd, for example: server { listen 80; server_name app.test.my-company.com; return 301 https://$host$request_uri; } server { listen 443 ssl http2; server_name app.test.my-company.com; # Configuration for SSL/TLS certificates ssl_certificate /app/certs/test.my-company.com.crt; ssl_certificate_key /app/certs/test.my-company.com.key; # Proxy headers proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; # Reverse proxy location / { resolver 127.0.0.11 valid=30s; # Docker DNS proxy_pass http://my-app:8080/; proxy_redirect default; } } However, suppose that the "my-app" container has a health check set up for it. What does that mean? Docker Swarm won't route any traffic to it until it's finished initialization. Except that it also means that DNS requests won't return anything for "my-app" until at least one healthcheck passes. That in of itself shouldn't be too problematic, right? Any web server surely would just log that one request as an error and return an error until eventually everything would work. Wrong. Nginx does throw the error and the request fails, but the ENTIRE web server goes down: nginx: [emerg] host not found in upstream "my-app" What does that mean? If you have 1 Nginx instance on the server that needs to proxy 10 apps in total, one app not being available will prevent the other 9 from working! Whoever thought that such defaults would be okay has reasoning that is beyond me, especially considering that the Caddy web server does something similar and will go down if 1 out of 10 sites fails to provision the SSL/TLS certificate, instead of letting the others work - completely opposite of what someone who cares about resiliency would want! But fear not, supposedly there's a solution! # Reverse proxy location / { resolver 127.0.0.11 valid=30s; # Docker DNS set $proxy_server my-app; proxy_pass http://$proxy_server:8080/; proxy_redirect default; } Except that, you know, that also doesn't work: nginx: [emerg] "proxy_redirect default" cannot be used with "proxy_pass" directive with variables Now, that's not an issue in my case, since i just jotted down the "proxy_redirect default;" value to remember what it does, but it's unfortunate to hear that it is neither ignored (since it's the default value which is functionally identical to it not being there), nor could other values be used: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_redirect http://nginx.org/en/docs/http/ngx_http_proxy_module.html#pro... In summary, it's nice that there are workaround for problems like that, but it's a bit odd that the web server is otherwise so quick to fall over and die, as well as the fact that the defaults aren't oriented towards maximum reliability and resilience. That said, i still do like Nginx and actually wrote about how to use it as a reverse proxy on my blog a while back: https://blog.kronis.dev/tutorials/how-to-use-nginx-to-proxy-your-front-end-and-back-end https://blog.kronis.dev/tutorials/how-to-use-nginx-to-proxy-... But at the same time, in situations where load isn't high enough for the web server to be the bottleneck, it's also helpful to evaluate the alternatives, notably: Apache2/httpd, Caddy and Traefik. It was actually pretty recently that Apache2/httpd got mod_md, which allows provisioning Let's Encrypt certificates out of the box, which is certainly promising: https://httpd.apache.org/docs/trunk/mod/mod_md.html https://httpd.apache.org/docs/trunk/mod/mod_md.html

bazzargh 5y ago

One we see a lot is this: https://trac.nginx.org/nginx/ticket/1151 https://trac.nginx.org/nginx/ticket/1151 worker_processes auto; in a kubernetes environment where the nginx container only has 1 cpu share, but is running on a 64 core node...this does the wrong thing and launches dozens of workers, which do nothing but eat memory. According to the bug above they've not fixed it because they can't tell the difference between this case and other cases where your cpus may be restricted. While that's true, other systems don't just ignore the problem: http://mail.openjdk.java.net/pipermail/hotspot-dev/2019-January/036120.html http://mail.openjdk.java.net/pipermail/hotspot-dev/2019-Janu... (we also commonly see issues with buffering, and proxy host header not being set; the latter is an issue when code moves from an environment with physical hosts - which don't care about the host header - to virtual ones)

xorcist 5y ago

There are some issues specific to nginx, and probably not the most common errors made. It's fundamentally hard to manipulate http requests without being knowing all forms paths and headers can take. Careless use of regexps in http routing is a common source of problems. Things like uri encoding, parameters as subdirectories and control characters is easy to get wrong. Treating user supplied headers as trusted is also easy to get wrong. It's not uncommon to see configurations which are remotely exploitable, which is another level of bad from what is described in the article.

jurschreuder 5y ago

Editing the nginx config until worked correctly has cost me so many days of my life. Luckily there are very good Golang alternatives for 99% of use cases now.

alberth 5y ago

What is the current recommended web server these days? Is it still nginx? I know caddy gets a lot of attention but always assumed it was for smaller/less-trafficked sites (maybe I'm wrong).

fraktl 5y ago
mholt 5y ago
Caddy is used by large enterprises and is processing hundreds of thousands of dollars of revenue per hour at one company I know of. You can definitely use it for high volume sites.

LinAGKar 5y ago

So add doesn't (always) add, and instead you have to repeat yourself all over the place? That's just stupid.

schwag09 5y ago

This article briefly mentions a very useful analysis tool for NGINX configuration: Gixy. It looks for the following misconfigurations[0]: - [ssrf] Server Side Request Forgery - [http_splitting] HTTP Splitting - [origins] Problems with referrer/origin validation - [add_header_redefinition] Redefining of response headers by "add_header" directive - [host_spoofing] Request's Host header forgery - [valid_referers] none in valid_referers - [add_header_multiline] Multiline response headers - [alias_traversal] Path traversal via misconfigured alias The alias traversal gotcha is one of the most pernicious I've seen. A single, seemingly innocuous '/' is the difference between a path traversal vulnerability or not. [0]: https://github.com/yandex/gixy#what-it-can-do https://github.com/yandex/gixy#what-it-can-do

bennettdams 5y ago

Isn't it ironic that even the writers of this article don't follow these rules in the examples on that page? There is "Mistake 3: Not Enabling Keepalive Connections to Upstream Servers", where it says: In the location{} block that forwards requests to an upstream group, include the following directives along with the proxy_pass directive: proxy_http_version 1.1; proxy_set_header "Connection" ""; Then, at "Mistake 10: Not Taking Advantage of Upstream Groups", the "location" block does not include these two directives, even though the "upstream" block contains a "keepalive" directive.

deleted 5y ago

[deleted]

deleted 5y ago

[deleted]

deleted 5y ago

[deleted]

kseifried 5y ago

I've assigned this the Global Security Database (GSD) ID GSD-2022-1000285 (https://raw.globalsecuritydatabase.org/GSD-2022-1000285 https://raw.globalsecuritydatabase.org/GSD-2022-1000285) and started a discussion about these informational entries at https://groups.google.com/a/groups.cloudsecurityalliance.org/g/gsd/c/t3l9DJXmN3A https://groups.google.com/a/groups.cloudsecurityalliance.org... TL;DR: if you run nginx I assume you want to know about these types of potential problems, so sticking it into the security database makes a lot of sense in my opinion for discoverability.

jdsleppy 5y ago

Nginx config seems like a bottomless pit. It gives me the painful feeling that no matter what I have done with it, there's some security or performance issue still there. Does this mean I should just stick with a more managed solution? Or is this a common feeling that I should just learn to live with? For my fun projects I use Caddy with mostly default settings.

freedomben 5y ago
I wouldn't be that worried about it. Sure there are ways to optimize it more and squeeze every drop of juice out the grape, but most of the time you're gonna be just fine. It might be the difference between running an extra app server or not. Now that said nginx website has a ton of great examples. You can usually copy/paste them into whatever you need pretty easily, and once it's done it's not likely to change. I would especially do this if nginx is your internet-facing load balancer and has multiple upstreams behind it. Plenty of good examples out there.

giobox 5y ago

nginx config increasingly feels like a bunch of foot guns waiting to go off for me. For smaller projects, I've really enjoyed Caddy's "best practice by default" approach to webserver/loadbalancer config where you can generally only choose to remove best practices, not try to build them up from scratch. Has SSL and LetsEncrypt enabled by default, etc. This avoids a lot of the pitfalls of incorrect webserver config altogether, keeps the configuration file really small and easy to read. I wish this style of config was more common. > https://caddyserver.com https://caddyserver.com

KronisLV 5y ago
I actually really like Caddy, though some of its defaults at least historically have been odd, such as responding with 200 where other web servers would respond with a 502 or something similar: https://caddy.community/t/why-does-caddy-return-an-empty-200-response-for-a-non-configured-host/7685 https://caddy.community/t/why-does-caddy-return-an-empty-200... Also, this is a bit of a personal preference, but v1 Caddy felt like maybe a bit easier to get started with than v2 Caddy, though sadly was abandoned. I'm sure that there were good reasons for doing it, but the few forks that started out from it with the intent to maintain it never really went everywhere, so v2 is the only possibility nowadays, unless you want to maintain it yourself or like dead software: https://github.com/WedgeServer/wedge https://github.com/WedgeServer/wedge Of course, that's not a criticism of Caddy itself, just how we as an industry sometimes need rewrites and that makes us keep up with the churn.
- francislavoie 5y ago
  The fact of the matter was that _servers aren't actually easy_. Caddy v1 tried to give the illusion that it is, but it just caused more problems, really. The complete rewrite for Caddy v2 was necessary to make Caddy flexible enough to solve like half or more of the open issues the Caddy repo had at that point. Flexibility and simplicity are often at odds. But Caddy tries to keep smart defaults in general. Also, any time someone brings up the malicious fork "wedge", it saddens me. That was done at literally the most stressful moment in the project's history, and it only made things worse for everyone. I really hope people learn to chill out and give maintainers a bit of time to breathe and respond instead of taking such hostile measures like that. (You can probably pretty easily dig up the Github issue with the discussion if you really care to see what went down. Sigh.)

cosmiccrisp 5y ago

I remember someone posting an article of common nginx config security mistakes but can't find it. Does anyone remember? I would like to read through that again as well.

WaxedChewbacca 5y ago
iancarroll 5y ago
Not a comprehensive article, but the worst security footgun, by far (IMO), is $uri. It’s completely unsafe to use $uri in basically any directives! You cannot redirect using it, proxy_pass using it, or you will have a bad time. https://reversebrain.github.io/2021/03/29/The-story-of-Nginx-and-uri-variable/ https://reversebrain.github.io/2021/03/29/The-story-of-Nginx...
- cosmiccrisp 5y ago
  Thanks! That is good info.

rr808 5y ago

Nginx would benefit from a simpler higher level instruction set or an official 2-5 recipes as I'm sure 99% of the implementations doing a few basic things and dont need the ability to screw it up.

binarymax 5y ago
Definitely. Been using it for years for many projects and it’s always the same needs: Let’s encrypt + static files + upstream reverse proxy. I had no idea about keepalive not being default - why isn’t this in all the examples and tutorials?
- ReactiveJelly 5y ago
  For that I use linuxserver's SWAG container. Let's Encrypt automates cert renewal, and SWAG automates having to remember how in the heck to set up LE.
- zhte415 5y ago
  I'm sure there' space in the tutorial area. A general point for $tech: If any reader is thinking of writing a tutorial, do check if it's been done before. Likely a grossly simplified example of how to do X while the person reading it has read upteen tutorials on how to do X, Y and Z but still doesn't know where to keep configuration files, or how anything interacts with anything else. Instead, think of writing Series Of Tutorials That Together Build Moderately Fully Fledged Thing Including Satellite Stuff Like, in this case, Nginx, or SMS, or why XX characters for an email address string, or whatever of the plethora of things to Make It Work are needed. Some books do this. Not a huge number though, certainly not compared to hit-and-run tutorials. Call them books. Call them Tutorial Super Series. I don't think the little more time invested would not generate outsized returns.

deleted 5y ago

[deleted]

The_rationalist 5y ago

cosmotic 5y ago

Nearly all of these are approximately "users did something they expected to work, but it didnt; shame on them" or "We chose a safe but mostly bad default, shame on users for not realizing it". I understand the rational for not changing old defaults in fear of breaking existing configs. However, maybe a release or two of deprecation or better optimized defaults under new names or a new (kinda gross but...) strict mode which is opt-in but is the new default for new instalations. 1. Warn the user on startup of potential misconfiguration 2. Allow error_log off to work, as users are expecting it to 3. Enable keep alive by default 4. Acknowledge the confusing syntax and find a better solution than blaming users 5. Warn the user on startup of impact of proxy_buffering off 6. See 4 7. See 1 8. See 1 9. Fix the proxy hash algorithm so that it hashes the whole address (why only first 3 octets?) or give it a new name; see also 1 10. Detect possible optimizations on startup and implement automatically; why ask the user to config something that can potentially be detected automatically?

josephg 5y ago
Absolutely. If you (nginx) know how your product should be configured, you should configure it that way out of the box. Don’t blame users. And if backward compatibility is important, take a leaf out of Rust’s book and make a nginx-config-version 2; directive or something. With this set, you can safely use the new defaults. Use it across all example configurations on the website and in distribution packages.
klaussilveira 5y ago
It is interesting to see nginx commit the same mistakes that led Apache to its demise.
BeFlatXIII 5y ago
> However, maybe a release or two of deprecation or better optimized defaults under new names or a new (kinda gross but...) strict mode which is opt-in but is the new default for new instalations. Something like this or a \version "2.12.0" command would be massively useful. The alternative is a sea of undisclosed footguns that are blindingly obvious once you know they exist but otherwise are well-camouflaged. Ok, the other alternative is for every future developer to brainlessly spam a long incantation of sensible values that they swear they'll learn about later just so a Fortune 500 early-adopter company can have smoother upgrades.

nemothekid 5y ago

I'm surprised the "set a resolver with a dns ttl" isn't on here; it's probably my only headache with nginx. If you have an upstream that you specify with a hostname, nginx will lookup the host, cache it, and never refresh it by default. I don't know why it happens to me so often, but I will set up nginx containers that proxy to some load balanced service (via DNS), and then one day the IPs will change and that service would break because I forgot to set resolver.

amdelamar 5y ago
This is my top issue with open-source nginx. Nginx plus has the solution though with its `resolver` directive. https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#resolve https://docs.nginx.com/nginx/admin-guide/load-balancer/http-...
- megous 5y ago
  Opensource nginx has the same resolver directive.
  - kzemek 5y ago
    But servers in upstream block don't have a 'resolve' setting in non-plus versions. 'resolver' still does something in the open source version, but requires some configuration contortions.
cyounkins 5y ago
This issue has bitten me multiple times. You must set the resolver AND use the domain name in a variable. See here: https://www.nginx.com/blog/dns-service-discovery-nginx-plus/ https://www.nginx.com/blog/dns-service-discovery-nginx-plus/ under "Setting the Domain Name in a Variable". The other options ignore the record TTL or require Nginx Plus. More details here: https://github.com/DmitryFillo/nginx-proxy-pitfalls https://github.com/DmitryFillo/nginx-proxy-pitfalls
- egberts1 5y ago

lokar 5y ago

Why would you pick nginx over envoy?

mrweasel 5y ago
They aren't really the same type of product. Envoy is mostly an alternative to something like HAProxy, or Træfik. You wouldn't pick Envoy over Nginx (or Apache) if you need a webserver, than wouldn't make sense. Envoy solves a very specific problem, Nginx solves a bunch of problems. Also if you're using Nginx everywhere else already, you might want to use a product you're comfortable with, even in the cases where Envoy could work for you. E.g. we use Apache almost everywhere, because performance is "good enough", and it allows us to use the same product for a lot of different client, due to the large feature set.
manigandham 5y ago
nginx is a reverse proxy and app server. It has all the major features to handle front/edge tasks (like TLS termination), serve static files, and run entire applications in-process given the correct modules. nginx has entire ecosystems around this with OpenResty and even runs the edge workload for Cloudflare's CDN and Algolia's search service. envoy is a L7 proxy focused on full duplex networking across multiple protocols with transparency and observability. It can also handle front/edge tasks although the configuration system is a magnitude more complex than nginx and it's also missing the app server abilities. If you use envoy already then it might be easier to deploy it to the edge as well, and it's often used that way in Kubernetes as both service-mesh and ingress - but if you just need typical reverse-proxy/app serving than nginx is still far simpler to setup and maintain.
deleted 5y ago
[deleted]
jcdavis 5y ago
Serves static content (which envoy still can't do AFAICT) Simpler to set up for a basic proxy. For more complex usage though, nginx is certainly outclassed (at least the OSS version)
mike_d 5y ago

jrtc27 5y ago

> The if directive is tricky to use, especially in location{} blocks. It often doesn’t do what you expect and can even cause segfaults. That's not user error. That's lazy programming and a bug. If your code segfaults because of user input you weren't expecting, that's on you as a programmer.

mdaniel 5y ago
It wouldn't surprise me if they consider that WONTFIX given they wrote a whole wiki page entitled "If is Evil": https://www.nginx.com/resources/wiki/start/topics/depth/ifisevil/ https://www.nginx.com/resources/wiki/start/topics/depth/ifis...
- jrtc27 5y ago
  Oh I know. Hardly inspires confidence; all of those examples are clear bugs declared features because of poor programming practices.
- Dylan16807 5y ago
  I can understand why certain directives don't work inside an if, even if segfault is dumb. Why are the empty if blocks breaking things outside of them?

egberts1 5y ago

We still cannot bind NGINX to a specific interface. sad

jesterson 5y ago
What do you mean here? If you are referring to network interface, why there is a problem to bind to it?
- sgjohnson 5y ago
  You can’t bind nginx to a specific network interface. You can bind it to a specific IP address though (and then, by extension, also to an interface, but you need to know the IP).
  - jesterson 5y ago
    That's right, although I am struggling to think of use cases where you may need to assign it to network interface with dynamic IP allocation (otherwise you'd know the IP beforehand).
    - sgjohnson 5y ago
      > although I am struggling to think of use cases where you may need to assign it to network interface with dynamic IP allocation Some failover configurations come to mind, in particular the one where the box doesn’t get an IP address on the said interface until the failover kicks in, but I don’t know why would anyone run this kind of a setup without NAT.
      - egberts1 5y ago
        I did that bind with DHCP server failover.
    - egberts1 5y ago
      My use case is simple. I have five disparate IP subnets whose firewall must NOW have to ascertain that ONLY a certain subnet can access this unbinded netdev … in case the NGINX (or any daemon) starts to sniff unwarranted IP subnet traffic. And yes, you can bind to a dynamic IP assigned netdev. And yes , I’ve had to replace NGINX with a web server that can do a netdev bind.

oxplot 5y ago

Traefik has been my go-to reverse proxy for quite a while now. The main advantages is its docker/k8s and native ACME support. You can configure services, config, routes, etc. via docker labels. When running under compose, this makes it very clean and convenient to maintain config. It also has plenty of useful middlewares and has a prometheus exporter for its internal stats.

mekster 5y ago
It's good we have many options to choose from. Apache, nginx, caddy, Traefik, HAProxy, lighttpd, Varnish, Envoy are likely to be the major ones?

vishnugupta 5y ago

Of late (~1 year) I've migrated to AWS ALB from Nginx and recommend it if you are on AWS ecosystem. Being a Layer-7 load balancer, ALB is (almost) feature compatible with Nginx. You can do route based, header based load balancing, terminate SSL and so on. For an internet facing app my preferred setup now a days is [CloudFront → ALB → application].

winrid 5y ago

The big one for me was the SSL cache size config generated by certbot combined with nchan. That caused so many production issues w/ many concurrent connections.

ranger207 5y ago

My biggest gripe with nginx is the documentation. I ran into an issue with proxy_buffering being off yet I was still getting 502 errors because the upstream was passing too large headers for the default 4k buffers. I naively assumed that with proxy buffering off that it would send data to the client immediately and, y'know, not buffer it, as [0] suggests. Turns out that turning proxy_buffering off doesn't turn proxy buffering completely off. While working on that issue, I also needed to set the proxy buffer size. Quick, what's the difference between proxy_buffer_size [1] and the size argument of proxy_buffers [2]? I still don't know, because the docs for each directive sound to me like they're restating the same exact thing. [0] https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering https://nginx.org/en/docs/http/ngx_http_proxy_module.html#pr... [1] https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size https://nginx.org/en/docs/http/ngx_http_proxy_module.html#pr... [2] https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers https://nginx.org/en/docs/http/ngx_http_proxy_module.html#pr...

Dylan16807 5y ago
Looking at that documentation has me even more confused. The article says "Proxy buffering means that NGINX stores the response from a server in internal buffers as it comes in, and doesn’t start sending data to the client until the entire response is buffered." But those parameters say nothing about waiting to send data to the client, and in fact some of them like proxy_busy_buffers_size imply the exact opposite.
- barrkel 5y ago
  Logically the purpose of proxy buffering is to prevent server death from clients holding connections open by reading the data very slowly (slowloris attack). But this doesn't require waiting for the entire upstream response before sending bytes to the client; merely that reading from upstream doesn't stop when the client is reading slowly. It looks like nginx does this while still preventing slowloris. The parameters mentioned look like they're about tuning the relative memory costs of sending down the first bit of data in a response, which may contain enough information on the client side to do something interesting. proxy_buffer_size looks like it's just the first buffer, while the size argument to proxy_buffers applies to all the buffers.

exciteabletom 5y ago

What are some alternatives to NGINX? Is there a maintained fork with a better config? I know of Apache, but that seems to be falling out of favour lately.

jimmySixDOF 5y ago
HAProxy is one and I'd like to see a breakdown of how these config issues either spillover or are dealt with differently in that environment.
- exciteabletom 5y ago
  Thanks so much for replying! Just looked at some example configs and it looks way slicker than nginx.
- mrweasel 5y ago
  There's an overlap in features between Nginx and HAProxy, but one isn't necessarily a replacement for the other. Nginx is a full webserver, HAProxy isn't a webserver.
ahmetkun 5y ago
> I know of Apache, but that seems to be falling out of favour lately. lately? more like for the last 10 years afaict.
j_autumn 5y ago
I like Caddy: https://caddyserver.com/ https://caddyserver.com/ Nice community and enough features for my (basic) use cases. The only thing that I don’t quite like is the logging format.
- exciteabletom 5y ago
  The automatic certificate renewal is a killer feature. Goodbye dodgy cronjobs!
mekster 5y ago