4 ms·

Fixng Nginx 502 Gateway Timeout with proxy_pass During Deployments

49 points by winrid 7y ago

viraptor 7y ago
Is this "fixing"? The proper fix would be to use a seamless deployment by either spawning a new vm with new version, then redirecting the traffic once it's ready. Or by doing the same with a new version of the app server on the same host and replacing the socket when ready. Otherwise, the requests will potentially wait a long time to be handled while the app starts up.
- winrid 7y ago
  In this case it's mostly for experiments and stuff. And spawning a new vm etc is slow for that kind of thing. This takes less than a second (I just merge to master and it deploys).
- winrid 7y ago
  also! I worked at a place where this was an issue for years. We eventually moved to Kubernetes, but it would have been nice to know this fix until then for the qa/dev environments where stuff was constantly deploying.
mehrdada 7y ago
Envoy is the new hotness in proxyland.
- winrid 7y ago
  Nice to know. However, if I think if I was going to redo everything I'd use Kubernetes+Istio?
  - hedwall 7y ago
    Istio uses Envoy for the data plane.
    - winrid 7y ago
      Oh cool. It shows I haven't dived very deeply into it yet. :)
kart23 7y ago
I've never had this problem when doing reverse proxies through nginx. Theres no delay when I restart node, the 502 goes away right after.
- winrid 7y ago
  You have to make a request while the upstream is down. So do the restart, then make a request right away. If your service restarts quickly it'll be hard to reproduce.
simplyinfinity 7y ago
To me the most appropriate way (when not dealing with kubernetes, but rather just vms) is to be able to mark specific server to not receive any new requests, wait for the requests to stop coming in and then deploy. IIS web server has this but not nginx :(
- pritambaral 7y ago
  I use HAProxy for this. The paid version of Nginx also has this.
- winrid 7y ago
  What about: https://www.nginx.com/blog/dynamic-reconfiguration-with-nginx-plus/ https://www.nginx.com/blog/dynamic-reconfiguration-with-ngin...
mfontani 7y ago
The article mentions nginx returning 502 when a backend is down during a deploy. This a problem in the cases where restarting the backend isn't atomic. If the backend is able to perform a "reload" (vs. a slower stop/start or restart) and/or replace the old instances with the new when they're up -- assuming that's an atomic operation -- nginx will not return a 502. If one is rapidly iterating and prefers having a somewhat cleaner stop/start, the advice can be helpful even though it masks/papers over a problem (the backend being down).
- winrid 7y ago
  Correct.
notyourday 7y ago
Detecting a failure and minimizing the request wreck that it causes is different from handling a deploy. The second one should cause no wrecks because your deploy management system should be able to tell your proxy layer to drain the target of deploy to allow the deploy to proceed when a target has no traffic

49 points by winrid 7y ago

Is this "fixing"? The proper fix would be to use a seamless deployment by either spawning a new vm with new version, then redirecting the traffic once it's ready. Or by doing the same with a new version of the app server on the same host and replacing the socket when ready. Otherwise, the requests will potentially wait a long time to be handled while the app starts up.

winrid 7y ago
In this case it's mostly for experiments and stuff. And spawning a new vm etc is slow for that kind of thing. This takes less than a second (I just merge to master and it deploys).
winrid 7y ago
also! I worked at a place where this was an issue for years. We eventually moved to Kubernetes, but it would have been nice to know this fix until then for the qa/dev environments where stuff was constantly deploying.

mehrdada 7y ago

Envoy is the new hotness in proxyland.

winrid 7y ago
Nice to know. However, if I think if I was going to redo everything I'd use Kubernetes+Istio?
- hedwall 7y ago
  Istio uses Envoy for the data plane.
  - winrid 7y ago
    Oh cool. It shows I haven't dived very deeply into it yet. :)

kart23 7y ago

I've never had this problem when doing reverse proxies through nginx. Theres no delay when I restart node, the 502 goes away right after.

winrid 7y ago
You have to make a request while the upstream is down. So do the restart, then make a request right away. If your service restarts quickly it'll be hard to reproduce.

simplyinfinity 7y ago

To me the most appropriate way (when not dealing with kubernetes, but rather just vms) is to be able to mark specific server to not receive any new requests, wait for the requests to stop coming in and then deploy. IIS web server has this but not nginx :(

pritambaral 7y ago
I use HAProxy for this. The paid version of Nginx also has this.
winrid 7y ago
What about: https://www.nginx.com/blog/dynamic-reconfiguration-with-nginx-plus/ https://www.nginx.com/blog/dynamic-reconfiguration-with-ngin...

mfontani 7y ago

The article mentions nginx returning 502 when a backend is down during a deploy. This a problem in the cases where restarting the backend isn't atomic. If the backend is able to perform a "reload" (vs. a slower stop/start or restart) and/or replace the old instances with the new when they're up -- assuming that's an atomic operation -- nginx will not return a 502. If one is rapidly iterating and prefers having a somewhat cleaner stop/start, the advice can be helpful even though it masks/papers over a problem (the backend being down).

winrid 7y ago
Correct.

notyourday 7y ago

Detecting a failure and minimizing the request wreck that it causes is different from handling a deploy. The second one should cause no wrecks because your deploy management system should be able to tell your proxy layer to drain the target of deploy to allow the deploy to proceed when a target has no traffic