4 ms·

I think this is why some techniques for doing rolling deploys set up a health check for the node which is configurable by the deploy process. That way, otherwi

I think this is why some techniques for doing rolling deploys set up a health check for the node which is configurable by the deploy process. That way, otherwise healthy nodes report their status as "down" while still accepting and finishing requests, but then get taken out of the HAProxy rotation without having to re-load the configuration. The new code is rolled out on that node, and the toggle is set to report "healthy" again, so HAProxy brings the node back into rotation.

nulltype 11y ago
This is what I've done in the past, not sure why we need to "hack TCP" to get zero downtime. As an additional thing I do rolling deploys where I just create an entirely new VM and add it before removing the old one. This just means that I don't have to recycle a node that might potentially have some state on it.
- tangled 11y ago
  The article is about restarting HAProxy without any downtime. HAProxy restarts are needed when adding new service instances or adjusting configuration options. This is a different and much harder problem than gracefully restarting load balanced service instances.
  - jolynch 11y ago
    As other posters have mentioned the solution in this link is actually suboptimal. I showed how the iptables solution leads to 1-3s latency for connections that establish during the restart but you can avoid it by getting a little more hardcore. See http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html http://engineeringblog.yelp.com/2015/04/true-zero-downtime-h... or http://inside.unbounce.com/product-dev/haproxy-reloads/ http://inside.unbounce.com/product-dev/haproxy-reloads/ for solutions that are even better than this one.
  - jsprogrammer 11y ago
    Why shouldn't HAProxy have an API for updating its configuration while running?
    - vidarh 11y ago
      It should, and it allows you to do some changes over a socket, but allowing complete updates with the current haproxy architecture is a lot of work and isn't there yet.
bbrazil 11y ago
That's the approach we take, documented at http://www.boxever.com/gracefully-shutting-down-haproxy-with-elb http://www.boxever.com/gracefully-shutting-down-haproxy-with... The challenge is that whether HAProxy's own support for graceful restarts works depends on which OS and kernel version you're running and what network socket features are supported, so ensuring that's all kosher is a bit of a challenge. I took the approach that it's simpler to do something that takes a bit longer without having to worry about all that.

nulltype 11y ago

This is what I've done in the past, not sure why we need to "hack TCP" to get zero downtime. As an additional thing I do rolling deploys where I just create an entirely new VM and add it before removing the old one. This just means that I don't have to recycle a node that might potentially have some state on it.

tangled 11y ago
The article is about restarting HAProxy without any downtime. HAProxy restarts are needed when adding new service instances or adjusting configuration options. This is a different and much harder problem than gracefully restarting load balanced service instances.
- jolynch 11y ago
  As other posters have mentioned the solution in this link is actually suboptimal. I showed how the iptables solution leads to 1-3s latency for connections that establish during the restart but you can avoid it by getting a little more hardcore. See http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html http://engineeringblog.yelp.com/2015/04/true-zero-downtime-h... or http://inside.unbounce.com/product-dev/haproxy-reloads/ http://inside.unbounce.com/product-dev/haproxy-reloads/ for solutions that are even better than this one.
- jsprogrammer 11y ago
  Why shouldn't HAProxy have an API for updating its configuration while running?
  - vidarh 11y ago
    It should, and it allows you to do some changes over a socket, but allowing complete updates with the current haproxy architecture is a lot of work and isn't there yet.

bbrazil 11y ago

That's the approach we take, documented at http://www.boxever.com/gracefully-shutting-down-haproxy-with-elb http://www.boxever.com/gracefully-shutting-down-haproxy-with... The challenge is that whether HAProxy's own support for graceful restarts works depends on which OS and kernel version you're running and what network socket features are supported, so ensuring that's all kosher is a bit of a challenge. I took the approach that it's simpler to do something that takes a bit longer without having to worry about all that.