5 ms·

This was more complicated that I hoped it would be. Is there any reason HAProxy couldn't simply add a way to reload it's conifg file while running? It already

This was more complicated that I hoped it would be. Is there any reason HAProxy couldn't simply add a way to reload it's conifg file while running? It already supports a bunch of commands to modify the config while it is running, via the Unix socket, but there's no way to pipe an entire config file to that socket. It wouldn't work for version upgrades, of course, but that's not really the main use case. Another option could be to start up a parallel HAProxy that listens to a different port. Then use iptables to route traffic to that port instead of the old one. Not sure if this would break connections, though, or if there is a way to prevent breaking connections.

duskwuff 12y ago
Another approach: have haproxy fork/exec a child, inheriting the same file descriptor for the listening socket, then have the child tell the parent when it should stop accepting connections on that socket. No idea how that works in practice, but it seems like a sound concept? :)
- lobster_johnson 12y ago
  Yes, that's how Unicorn works. But HAProxy isn't designed to inherit another process's current state (see this other comment: https://news.ycombinator.com/item?id=9371064 https://news.ycombinator.com/item?id=9371064), and it's probably quite complicated and error-prone. If you're going down that route, it's probably much easier to write the code required to simply re-read the config and apply a diff to the internal data structures.
  - deleted 12y ago
    [deleted]
- ekimekim 12y ago
  A less fragile approach would be for the old process to simply pass the listen fd to the new process over a unix socket. You could also pass any active connections in this way.
toong 12y ago
I think you can use -m state --state ESTABLISHED,RELATED to match existing tcp flows. But that builds on conntrack, you could get in trouble if this HA proxy is public facing and handles a gazillion requests ?
- lobster_johnson 12y ago
  Probably, but realistically, it's probably fine for most shops.
  - toong 12y ago
    Most shops don't handle a gazillion requests. And if you don't handle a gazillion requests, you don't really care about 20ms downtime while HA proxy is reloading ?
moe 12y ago
Another option could be to start up a parallel HAProxy that listens to a different port. That's how we do it and the script isn't even complicated. Use iptables to shift the traffic to the new instance, wait for the old instance to drain (netstat is your friend), and kill it. Much respect to the effort that went into the yelp solution, but it does feel a bit brittle and over-engineered for a such a simple problem.
- snowman17 12y ago
  The solution you mention has some tradeoffs. Imagine that you have hundreds of services each of which have a port and have some long running connections. With your solution you can only restart the load balancer at the rate of the longest running connections or whenever ports run out, whichever comes first. Also then you'd have to maintain the logic to do the port mapping, iptables switching, draining, etc ... The post mentions they considered this option and they claim that they were worried about engineering risk; seems reasonable to be worried about that.
  - moe 12y ago
    With your solution you can only restart the load balancer at the rate of the longest running connections Well, perhaps I should have provided more detail. Our ansible deployment script actually counts the port up until it finds a free one (we keep a port-range reserved for this purpose). So if there are multiple changes in rapid succession then more than two haproxy instances may dangle around for a while. The "port discovery" is a shell-task that registers the port to be used as a variable, which is then used by the templates of the haproxy and iptables roles. The cleanup is done by a 15min-cronjob which kills all haproxy instances that have no connections in netstat and don't match the haproxy pidfile.
- deleted 12y ago
  [deleted]
  - deleted 12y ago
    [deleted]
- 12y ago
snowman17 12y ago
There has been ongoing work in HAProxy for many years to support zero downtime reloads via a few different potential mechanisms (file descriptor passing, a socket server, etc ...). Unfortunately it turns out that it is really hard given the architecture of HAProxy. That being said, I'm sure patches are always welcome. The post does mention that they did consider something similar to what you are suggesting with the multiple HAProxy instances but decided against it due to engineering uncertainties. Could be that they just overestimated how hard it would be.
jolynch 12y ago
This was more complicated that I hoped it would be I also hoped that it would be less complicated, but after prototyping a few different options, many of which I mention in the blog post (e.g. a patch to do fd passing, huptime, or running multiple parallel HAProxies), I decided that this was the lowest risk given the engineering constraints that we had. Also I wasn't entirely sure of an implementation plan for how to cleanly do port mappings for the hundreds of services. For example our service provisioning system adds, change, move, or removes services in various Yelp environments dynamically, so the port mapping logic would have to keep sync with that. It is certainly possible but I stand by my concern. To put this in perspective, I got this working in production at Yelp in about two or three days, and it has not had a single issue since. With this solution we apply some TC rules and some iptables rules once, and then the act of restarting is just a set of TC commands. It solves all of our use cases and does so without significant infrastructure that someone else has to maintain two years from now. If a month from now HAProxy releases native zero downtime restarts, I don't feel bad about throwing all of this away.

duskwuff 12y ago

Another approach: have haproxy fork/exec a child, inheriting the same file descriptor for the listening socket, then have the child tell the parent when it should stop accepting connections on that socket. No idea how that works in practice, but it seems like a sound concept? :)

lobster_johnson 12y ago
Yes, that's how Unicorn works. But HAProxy isn't designed to inherit another process's current state (see this other comment: https://news.ycombinator.com/item?id=9371064 https://news.ycombinator.com/item?id=9371064), and it's probably quite complicated and error-prone. If you're going down that route, it's probably much easier to write the code required to simply re-read the config and apply a diff to the internal data structures.
- deleted 12y ago
  [deleted]
ekimekim 12y ago
A less fragile approach would be for the old process to simply pass the listen fd to the new process over a unix socket. You could also pass any active connections in this way.

toong 12y ago

I think you can use -m state --state ESTABLISHED,RELATED to match existing tcp flows. But that builds on conntrack, you could get in trouble if this HA proxy is public facing and handles a gazillion requests ?

lobster_johnson 12y ago
Probably, but realistically, it's probably fine for most shops.
- toong 12y ago
  Most shops don't handle a gazillion requests. And if you don't handle a gazillion requests, you don't really care about 20ms downtime while HA proxy is reloading ?

moe 12y ago

Another option could be to start up a parallel HAProxy that listens to a different port. That's how we do it and the script isn't even complicated. Use iptables to shift the traffic to the new instance, wait for the old instance to drain (netstat is your friend), and kill it. Much respect to the effort that went into the yelp solution, but it does feel a bit brittle and over-engineered for a such a simple problem.

snowman17 12y ago
The solution you mention has some tradeoffs. Imagine that you have hundreds of services each of which have a port and have some long running connections. With your solution you can only restart the load balancer at the rate of the longest running connections or whenever ports run out, whichever comes first. Also then you'd have to maintain the logic to do the port mapping, iptables switching, draining, etc ... The post mentions they considered this option and they claim that they were worried about engineering risk; seems reasonable to be worried about that.
- moe 12y ago
  With your solution you can only restart the load balancer at the rate of the longest running connections Well, perhaps I should have provided more detail. Our ansible deployment script actually counts the port up until it finds a free one (we keep a port-range reserved for this purpose). So if there are multiple changes in rapid succession then more than two haproxy instances may dangle around for a while. The "port discovery" is a shell-task that registers the port to be used as a variable, which is then used by the templates of the haproxy and iptables roles. The cleanup is done by a 15min-cronjob which kills all haproxy instances that have no connections in netstat and don't match the haproxy pidfile.
deleted 12y ago
[deleted]
- deleted 12y ago
  [deleted]
12y ago

snowman17 12y ago

There has been ongoing work in HAProxy for many years to support zero downtime reloads via a few different potential mechanisms (file descriptor passing, a socket server, etc ...). Unfortunately it turns out that it is really hard given the architecture of HAProxy. That being said, I'm sure patches are always welcome. The post does mention that they did consider something similar to what you are suggesting with the multiple HAProxy instances but decided against it due to engineering uncertainties. Could be that they just overestimated how hard it would be.

jolynch 12y ago

This was more complicated that I hoped it would be I also hoped that it would be less complicated, but after prototyping a few different options, many of which I mention in the blog post (e.g. a patch to do fd passing, huptime, or running multiple parallel HAProxies), I decided that this was the lowest risk given the engineering constraints that we had. Also I wasn't entirely sure of an implementation plan for how to cleanly do port mappings for the hundreds of services. For example our service provisioning system adds, change, move, or removes services in various Yelp environments dynamically, so the port mapping logic would have to keep sync with that. It is certainly possible but I stand by my concern. To put this in perspective, I got this working in production at Yelp in about two or three days, and it has not had a single issue since. With this solution we apply some TC rules and some iptables rules once, and then the act of restarting is just a set of TC commands. It solves all of our use cases and does so without significant infrastructure that someone else has to maintain two years from now. If a month from now HAProxy releases native zero downtime restarts, I don't feel bad about throwing all of this away.