4 ms·

My org moved off nginx for haproxy after we learned that (at the time, maybe it changed) reloading an nginx config, even if done gracefully through kernel signa

My org moved off nginx for haproxy after we learned that (at the time, maybe it changed) reloading an nginx config, even if done gracefully through kernel signals, would drop existing connections, where haproxy could handle it gracefully. That was a fun week of diving in to some C code looking for why it was behaving that way.

nullify88 3y ago
How did you come to that conclusion? I always believed a reload spawned new workers and let the old one drain off.
- CogitoCogito 3y ago
  Yes I reload nginx all the time and it doesn’t drop connections. I just use the debian nginx package. Not sure what the gp is talking about.
  - nvarsj 3y ago
    Nginx abruptly drops http/1.1 persistent connections on reloads. This has been an issue forever and Maxim refused to ever fix it, saying it was to spec (yes it was, but there are better ways to deal with it). It’s a reason why many large, modern infra deployments have moved away from nginx.
    - xorcist 3y ago
      It doesn't drop it, it's just not persistent on reload, isn't that what you mean? Actually dropping a connection mid-request is something I haven't seen nginx (or indeed Apache) do for many years despite doing some weird things with it. I can see where you're coming from, but it's not unreasonable behaviour, is it? Connections needs to migrated over to the new worker and that's how all major servers do it. If that's a problem then maybe something designed as proxy only instead of a real server is the way to go?
      - nullify88 3y ago
        Isn't the typical behaviour of an application to re-establish the persistent connection on demand? I wonder what the requirement is to have these persistent with no timeout.
        Too 3y ago
        Yep. Persistent connections are bound to fail sooner or later anyway, so a robust application should have its own recovery.
      - nvarsj 3y ago
        It doesn't drop mid request. But it closes the TCP socket abruptly after any in flight requests are completed. Clients have no idea the connection is closed, and try to reuse it and get back a RST. In heavily dynamic environments where nginx reloads happen frequently, it leads to large amounts of RSTs/broken connections and high error rates (you can't necessarily auto-retry a POST, a RST could mean anything). The sane approach is connection draining - you send a `connection: close` response header on the old worker, then finally remove any remaining idle connections at the end of the drain. In http/2 it's not an issue as it has a way for the server to explicitly say the connection is closed.
        eddieroger 3y ago
        I'm late to return to the thread, but this was the exact scenario we hit. We had it behind a CDN as well as behind an L4 load balancer for some very high volume internal consumers, and when it would just blast back RST packets, the consumers would freak out and break connection, returning errors on their end that weren't matched in our logs, unless we were lucky and got a 499 (now Maxim can talk about standards). As a general purpose reverse proxy for many clients on the Internet, I'm sure that's fine, but in our use case this made nginx unpredictable and no longer desirable.
        xorcist 3y ago
        What you describe is basically how persistent http works, is it not? Even a persistent connection terminates at some point. Which web server does not work like that? I guess you could send the connection header on draining, but anything less than what the big servers do is bound to cause some compatibility problem with some niche client somewhere. I can certainly see why a web server with millions of installs would be reluctant to change bevaviour, even if it is within spec. I can only guess at the use case here, but maybe something designed from the start as a stateless proxy and not a general purpose web server would be a better fit.
jamespwilliams 3y ago
nginx supports graceful reloading and I’m pretty sure it has for a very long time - there are references to it in the changelog from 2005 https://nginx.org/en/docs/control.html https://nginx.org/en/docs/control.html
zetsurin 3y ago
we went in the opposite direction, not because haproxy was bad, just because nginx had a simpler config, and i think we were paying for haproxy but don't pay for nginx. all that said, neither drops existing connections on reload
yakshaving_jgt 3y ago
Another issue with nginx IIRC is that it allows HTTP request smuggling, which is a critical security vulnerability.
- callahad 3y ago
  That's been fixed for years. The CVE I can find was resolved in 1.17.7 (Dec 2019), and further hardening was applied in 1.21.1 (Jul 2021).
  - yakshaving_jgt 3y ago
    Can nginx send requests upstream over HTTP/2? I see this question has remained unanswered for a couple of years. https://security.stackexchange.com/questions/257823/what-are-nginx-reverseproxy-users-doing-to-prevent-http-request-smuggling https://security.stackexchange.com/questions/257823/what-are...
    - callahad 3y ago
      It cannot. There's more detailed reasoning at https://trac.nginx.org/nginx/ticket/923 https://trac.nginx.org/nginx/ticket/923, but the tl;dr is: > nginx is already good at mitigating HTTP desync / request smuggling attacks, even without using HTTP/2 to backends. In particular because it normalizes Content-Length and Transfer-Encoding while routing requests (and also does not reuse connections to backend servers unless explicitly configured to do so)

nullify88 3y ago

How did you come to that conclusion? I always believed a reload spawned new workers and let the old one drain off.

CogitoCogito 3y ago
Yes I reload nginx all the time and it doesn’t drop connections. I just use the debian nginx package. Not sure what the gp is talking about.
- nvarsj 3y ago
  Nginx abruptly drops http/1.1 persistent connections on reloads. This has been an issue forever and Maxim refused to ever fix it, saying it was to spec (yes it was, but there are better ways to deal with it). It’s a reason why many large, modern infra deployments have moved away from nginx.
  - xorcist 3y ago
    It doesn't drop it, it's just not persistent on reload, isn't that what you mean? Actually dropping a connection mid-request is something I haven't seen nginx (or indeed Apache) do for many years despite doing some weird things with it. I can see where you're coming from, but it's not unreasonable behaviour, is it? Connections needs to migrated over to the new worker and that's how all major servers do it. If that's a problem then maybe something designed as proxy only instead of a real server is the way to go?
    - nullify88 3y ago
      Isn't the typical behaviour of an application to re-establish the persistent connection on demand? I wonder what the requirement is to have these persistent with no timeout.
      - Too 3y ago
        Yep. Persistent connections are bound to fail sooner or later anyway, so a robust application should have its own recovery.
    - nvarsj 3y ago
      It doesn't drop mid request. But it closes the TCP socket abruptly after any in flight requests are completed. Clients have no idea the connection is closed, and try to reuse it and get back a RST. In heavily dynamic environments where nginx reloads happen frequently, it leads to large amounts of RSTs/broken connections and high error rates (you can't necessarily auto-retry a POST, a RST could mean anything). The sane approach is connection draining - you send a `connection: close` response header on the old worker, then finally remove any remaining idle connections at the end of the drain. In http/2 it's not an issue as it has a way for the server to explicitly say the connection is closed.
      - eddieroger 3y ago
        I'm late to return to the thread, but this was the exact scenario we hit. We had it behind a CDN as well as behind an L4 load balancer for some very high volume internal consumers, and when it would just blast back RST packets, the consumers would freak out and break connection, returning errors on their end that weren't matched in our logs, unless we were lucky and got a 499 (now Maxim can talk about standards). As a general purpose reverse proxy for many clients on the Internet, I'm sure that's fine, but in our use case this made nginx unpredictable and no longer desirable.
      - xorcist 3y ago
        What you describe is basically how persistent http works, is it not? Even a persistent connection terminates at some point. Which web server does not work like that? I guess you could send the connection header on draining, but anything less than what the big servers do is bound to cause some compatibility problem with some niche client somewhere. I can certainly see why a web server with millions of installs would be reluctant to change bevaviour, even if it is within spec. I can only guess at the use case here, but maybe something designed from the start as a stateless proxy and not a general purpose web server would be a better fit.

jamespwilliams 3y ago

nginx supports graceful reloading and I’m pretty sure it has for a very long time - there are references to it in the changelog from 2005 https://nginx.org/en/docs/control.html https://nginx.org/en/docs/control.html

zetsurin 3y ago

we went in the opposite direction, not because haproxy was bad, just because nginx had a simpler config, and i think we were paying for haproxy but don't pay for nginx. all that said, neither drops existing connections on reload

yakshaving_jgt 3y ago

Another issue with nginx IIRC is that it allows HTTP request smuggling, which is a critical security vulnerability.

callahad 3y ago
That's been fixed for years. The CVE I can find was resolved in 1.17.7 (Dec 2019), and further hardening was applied in 1.21.1 (Jul 2021).
- yakshaving_jgt 3y ago
  Can nginx send requests upstream over HTTP/2? I see this question has remained unanswered for a couple of years. https://security.stackexchange.com/questions/257823/what-are-nginx-reverseproxy-users-doing-to-prevent-http-request-smuggling https://security.stackexchange.com/questions/257823/what-are...
  - callahad 3y ago
    It cannot. There's more detailed reasoning at https://trac.nginx.org/nginx/ticket/923 https://trac.nginx.org/nginx/ticket/923, but the tl;dr is: > nginx is already good at mitigating HTTP desync / request smuggling attacks, even without using HTTP/2 to backends. In particular because it normalizes Content-Length and Transfer-Encoding while routing requests (and also does not reuse connections to backend servers unless explicitly configured to do so)