5 ms·

How did you come to that conclusion? I always believed a reload spawned new workers and let the old one drain off.

CogitoCogito 3y ago
Yes I reload nginx all the time and it doesn’t drop connections. I just use the debian nginx package. Not sure what the gp is talking about.
- nvarsj 3y ago
  Nginx abruptly drops http/1.1 persistent connections on reloads. This has been an issue forever and Maxim refused to ever fix it, saying it was to spec (yes it was, but there are better ways to deal with it). It’s a reason why many large, modern infra deployments have moved away from nginx.
  - xorcist 3y ago
    It doesn't drop it, it's just not persistent on reload, isn't that what you mean? Actually dropping a connection mid-request is something I haven't seen nginx (or indeed Apache) do for many years despite doing some weird things with it. I can see where you're coming from, but it's not unreasonable behaviour, is it? Connections needs to migrated over to the new worker and that's how all major servers do it. If that's a problem then maybe something designed as proxy only instead of a real server is the way to go?
    - nullify88 3y ago
      Isn't the typical behaviour of an application to re-establish the persistent connection on demand? I wonder what the requirement is to have these persistent with no timeout.
      - Too 3y ago
        Yep. Persistent connections are bound to fail sooner or later anyway, so a robust application should have its own recovery.
    - nvarsj 3y ago
      It doesn't drop mid request. But it closes the TCP socket abruptly after any in flight requests are completed. Clients have no idea the connection is closed, and try to reuse it and get back a RST. In heavily dynamic environments where nginx reloads happen frequently, it leads to large amounts of RSTs/broken connections and high error rates (you can't necessarily auto-retry a POST, a RST could mean anything). The sane approach is connection draining - you send a `connection: close` response header on the old worker, then finally remove any remaining idle connections at the end of the drain. In http/2 it's not an issue as it has a way for the server to explicitly say the connection is closed.
      - eddieroger 3y ago
        I'm late to return to the thread, but this was the exact scenario we hit. We had it behind a CDN as well as behind an L4 load balancer for some very high volume internal consumers, and when it would just blast back RST packets, the consumers would freak out and break connection, returning errors on their end that weren't matched in our logs, unless we were lucky and got a 499 (now Maxim can talk about standards). As a general purpose reverse proxy for many clients on the Internet, I'm sure that's fine, but in our use case this made nginx unpredictable and no longer desirable.
      - xorcist 3y ago
        What you describe is basically how persistent http works, is it not? Even a persistent connection terminates at some point. Which web server does not work like that? I guess you could send the connection header on draining, but anything less than what the big servers do is bound to cause some compatibility problem with some niche client somewhere. I can certainly see why a web server with millions of installs would be reluctant to change bevaviour, even if it is within spec. I can only guess at the use case here, but maybe something designed from the start as a stateless proxy and not a general purpose web server would be a better fit.

Yes I reload nginx all the time and it doesn’t drop connections. I just use the debian nginx package. Not sure what the gp is talking about.

nvarsj 3y ago
Nginx abruptly drops http/1.1 persistent connections on reloads. This has been an issue forever and Maxim refused to ever fix it, saying it was to spec (yes it was, but there are better ways to deal with it). It’s a reason why many large, modern infra deployments have moved away from nginx.
- xorcist 3y ago
  It doesn't drop it, it's just not persistent on reload, isn't that what you mean? Actually dropping a connection mid-request is something I haven't seen nginx (or indeed Apache) do for many years despite doing some weird things with it. I can see where you're coming from, but it's not unreasonable behaviour, is it? Connections needs to migrated over to the new worker and that's how all major servers do it. If that's a problem then maybe something designed as proxy only instead of a real server is the way to go?
  - nullify88 3y ago
    Isn't the typical behaviour of an application to re-establish the persistent connection on demand? I wonder what the requirement is to have these persistent with no timeout.
    - Too 3y ago
      Yep. Persistent connections are bound to fail sooner or later anyway, so a robust application should have its own recovery.
  - nvarsj 3y ago
    It doesn't drop mid request. But it closes the TCP socket abruptly after any in flight requests are completed. Clients have no idea the connection is closed, and try to reuse it and get back a RST. In heavily dynamic environments where nginx reloads happen frequently, it leads to large amounts of RSTs/broken connections and high error rates (you can't necessarily auto-retry a POST, a RST could mean anything). The sane approach is connection draining - you send a `connection: close` response header on the old worker, then finally remove any remaining idle connections at the end of the drain. In http/2 it's not an issue as it has a way for the server to explicitly say the connection is closed.
    - eddieroger 3y ago
      I'm late to return to the thread, but this was the exact scenario we hit. We had it behind a CDN as well as behind an L4 load balancer for some very high volume internal consumers, and when it would just blast back RST packets, the consumers would freak out and break connection, returning errors on their end that weren't matched in our logs, unless we were lucky and got a 499 (now Maxim can talk about standards). As a general purpose reverse proxy for many clients on the Internet, I'm sure that's fine, but in our use case this made nginx unpredictable and no longer desirable.
    - xorcist 3y ago
      What you describe is basically how persistent http works, is it not? Even a persistent connection terminates at some point. Which web server does not work like that? I guess you could send the connection header on draining, but anything less than what the big servers do is bound to cause some compatibility problem with some niche client somewhere. I can certainly see why a web server with millions of installs would be reluctant to change bevaviour, even if it is within spec. I can only guess at the use case here, but maybe something designed from the start as a stateless proxy and not a general purpose web server would be a better fit.