3 ms·

It doesn't drop mid request. But it closes the TCP socket abruptly after any in flight requests are completed. Clients have no idea the connection is closed, an

by nvarsj 3y ago

It doesn't drop mid request. But it closes the TCP socket abruptly after any in flight requests are completed. Clients have no idea the connection is closed, and try to reuse it and get back a RST. In heavily dynamic environments where nginx reloads happen frequently, it leads to large amounts of RSTs/broken connections and high error rates (you can't necessarily auto-retry a POST, a RST could mean anything). The sane approach is connection draining - you send a `connection: close` response header on the old worker, then finally remove any remaining idle connections at the end of the drain. In http/2 it's not an issue as it has a way for the server to explicitly say the connection is closed.

eddieroger 3y ago
I'm late to return to the thread, but this was the exact scenario we hit. We had it behind a CDN as well as behind an L4 load balancer for some very high volume internal consumers, and when it would just blast back RST packets, the consumers would freak out and break connection, returning errors on their end that weren't matched in our logs, unless we were lucky and got a 499 (now Maxim can talk about standards). As a general purpose reverse proxy for many clients on the Internet, I'm sure that's fine, but in our use case this made nginx unpredictable and no longer desirable.
xorcist 3y ago
What you describe is basically how persistent http works, is it not? Even a persistent connection terminates at some point. Which web server does not work like that? I guess you could send the connection header on draining, but anything less than what the big servers do is bound to cause some compatibility problem with some niche client somewhere. I can certainly see why a web server with millions of installs would be reluctant to change bevaviour, even if it is within spec. I can only guess at the use case here, but maybe something designed from the start as a stateless proxy and not a general purpose web server would be a better fit.

by nvarsj 3y ago

eddieroger 3y ago

I'm late to return to the thread, but this was the exact scenario we hit. We had it behind a CDN as well as behind an L4 load balancer for some very high volume internal consumers, and when it would just blast back RST packets, the consumers would freak out and break connection, returning errors on their end that weren't matched in our logs, unless we were lucky and got a 499 (now Maxim can talk about standards). As a general purpose reverse proxy for many clients on the Internet, I'm sure that's fine, but in our use case this made nginx unpredictable and no longer desirable.

xorcist 3y ago

What you describe is basically how persistent http works, is it not? Even a persistent connection terminates at some point. Which web server does not work like that? I guess you could send the connection header on draining, but anything less than what the big servers do is bound to cause some compatibility problem with some niche client somewhere. I can certainly see why a web server with millions of installs would be reluctant to change bevaviour, even if it is within spec. I can only guess at the use case here, but maybe something designed from the start as a stateless proxy and not a general purpose web server would be a better fit.