6 ms·

This release contains a neat feature: you can now bind HAProxy to a specific FD opened by its parent process. This means that you can babysit your HAProxy proc

by coops 12y ago

This release contains a neat feature: you can now bind HAProxy to a specific FD opened by its parent process. This means that you can babysit your HAProxy processes underneath a parent process that opens ports and get hitless HAProxy restarts, which I've long desired.

kawsper 12y ago
Wouldn't you get the same with a HAProxy reload with a SIGTERM?
- radoslawc 12y ago
  No, not really, for short period of time you've got state where previously configured instance is not working already and future one is not working yet.
  - coops 12y ago
    This is generally correct. In particular, when the HAProxy process is stopped/restarted there is a brief period during which the port is not bound by either process. (If the new process isn't able to get the socket when it boots it will sleep ~XXms, then try to bind/listen in a loop until it gets it or a retry threshold is hit.) During this time the kernel will reject incoming connections to the HAProxy port, so you are in danger of dropping incoming requests on the ground.
  - kawsper 12y ago
    Oh, you are right, after searching a little on the subject it looks like some people are doing things like this: iptables -I INPUT -p tcp -m multiport —dports 80,443 —syn -j DROP && sleep 0.5 && \ /etc/init.d/haproxy reload; iptables -D INPUT -p -tcp -m multiport —dports 80,443 —syn -j DROP Source: https://medium.com/@Drew_Stokes/actual-zero-downtime-with-haproxy-18318578fde6 https://medium.com/@Drew_Stokes/actual-zero-downtime-with-ha...
    - coops 12y ago
      I feel I must point out that this relies on your clients to retry packets that iptables drops, so at the least they'll have a slower experience than they otherwise would. I believe that this will also break requests that are in flight when the reload happens, if they have sent partial data.
      - kawsper 12y ago
        Yeah, it feels like a dirty, dirty hack. The way I understood it is that the requests in flight would retry the same way as the new connections when the first iptables rule is applied.
        coops 12y ago
        I think what will actually happen to requests in flight is: - partial data received by old HAProxy is lost as old HAProxy exits - new HAProxy comes online, binds to port, receives fd - iptables rule removed. new HAProxy starts receiving new requests - in-flight requests from the old HAProxy are timed out by the kernel (TCP RST) as nothing is there to read request data from the old fd or send response data. So I think this is actually "worse" in some sense than the other retry behavior since it's not recovered inside the same TCP session but instead forces the client to open a new TCP session.
daurnimator 12y ago
This is great for systemd integration! http://0pointer.de/blog/projects/socket-activation.html http://0pointer.de/blog/projects/socket-activation.html
ebroder 12y ago
Have you looked at einhorn? We've been running HAProxy under einhorn for a while now, using something like https://gist.github.com/ebroder/36b2f4f3aa210b9d9f3d https://gist.github.com/ebroder/36b2f4f3aa210b9d9f3d to translate between HAProxy's signalling mechanisms and einhorn's signalling mechanisms.
- coops 12y ago
  Er, yes, hi Evan! Cooper here. I believe either you or Andy originally pointed this interesting HAProxy restart behavior out to me, in the context of explaining why you wrote Einhorn.
ubercore 12y ago
This will also allow HAProxy to be brought into the fold if you're using Circus (http://circus.readthedocs.org/en/0.11.1/ http://circus.readthedocs.org/en/0.11.1/) for process monitoring. Probably not a big win in reliability, but I bet there are plenty of other things you can do with that (like parent mentioned)
indutny 12y ago
The bud ( https://github.com/indutny/bud https://github.com/indutny/bud ) does support this kind of hot config reloads and process restarts. It's just starting new worker processes on SIGHUP, and let old workers wait until all their connections will be closed. This is basically done by moving the balancing between workers into the master, instead of calling `accept()` concurrently from the workers.

by coops 12y ago

kawsper 12y ago

Wouldn't you get the same with a HAProxy reload with a SIGTERM?

radoslawc 12y ago
No, not really, for short period of time you've got state where previously configured instance is not working already and future one is not working yet.
- coops 12y ago
  This is generally correct. In particular, when the HAProxy process is stopped/restarted there is a brief period during which the port is not bound by either process. (If the new process isn't able to get the socket when it boots it will sleep ~XXms, then try to bind/listen in a loop until it gets it or a retry threshold is hit.) During this time the kernel will reject incoming connections to the HAProxy port, so you are in danger of dropping incoming requests on the ground.
- kawsper 12y ago
  Oh, you are right, after searching a little on the subject it looks like some people are doing things like this: iptables -I INPUT -p tcp -m multiport —dports 80,443 —syn -j DROP && sleep 0.5 && \ /etc/init.d/haproxy reload; iptables -D INPUT -p -tcp -m multiport —dports 80,443 —syn -j DROP Source: https://medium.com/@Drew_Stokes/actual-zero-downtime-with-haproxy-18318578fde6 https://medium.com/@Drew_Stokes/actual-zero-downtime-with-ha...
  - coops 12y ago
    I feel I must point out that this relies on your clients to retry packets that iptables drops, so at the least they'll have a slower experience than they otherwise would. I believe that this will also break requests that are in flight when the reload happens, if they have sent partial data.
    - kawsper 12y ago
      Yeah, it feels like a dirty, dirty hack. The way I understood it is that the requests in flight would retry the same way as the new connections when the first iptables rule is applied.
      - coops 12y ago
        I think what will actually happen to requests in flight is: - partial data received by old HAProxy is lost as old HAProxy exits - new HAProxy comes online, binds to port, receives fd - iptables rule removed. new HAProxy starts receiving new requests - in-flight requests from the old HAProxy are timed out by the kernel (TCP RST) as nothing is there to read request data from the old fd or send response data. So I think this is actually "worse" in some sense than the other retry behavior since it's not recovered inside the same TCP session but instead forces the client to open a new TCP session.

daurnimator 12y ago

This is great for systemd integration! http://0pointer.de/blog/projects/socket-activation.html http://0pointer.de/blog/projects/socket-activation.html

ebroder 12y ago

Have you looked at einhorn? We've been running HAProxy under einhorn for a while now, using something like https://gist.github.com/ebroder/36b2f4f3aa210b9d9f3d https://gist.github.com/ebroder/36b2f4f3aa210b9d9f3d to translate between HAProxy's signalling mechanisms and einhorn's signalling mechanisms.

coops 12y ago
Er, yes, hi Evan! Cooper here. I believe either you or Andy originally pointed this interesting HAProxy restart behavior out to me, in the context of explaining why you wrote Einhorn.

ubercore 12y ago

This will also allow HAProxy to be brought into the fold if you're using Circus (http://circus.readthedocs.org/en/0.11.1/ http://circus.readthedocs.org/en/0.11.1/) for process monitoring. Probably not a big win in reliability, but I bet there are plenty of other things you can do with that (like parent mentioned)

indutny 12y ago

The bud ( https://github.com/indutny/bud https://github.com/indutny/bud ) does support this kind of hot config reloads and process restarts. It's just starting new worker processes on SIGHUP, and let old workers wait until all their connections will be closed. This is basically done by moving the balancing between workers into the master, instead of calling `accept()` concurrently from the workers.