4 ms·

I'm moving away from HAProxy. The lack of zero downtime config reloads is consistently causing problems, especially in a high reload environment (i.e. using bam

by fidget 11y ago

I'm moving away from HAProxy. The lack of zero downtime config reloads is consistently causing problems, especially in a high reload environment (i.e. using bamboo [0]). One team deploys an application that goes into a restart loop, changing our routing table constantly? Well there goes our 99th percentile out the window. That said, would prefer not to be using nginx, so if anyone has any recommendations for high quality zero-downtime-reloadable HTTP proxies, I'd be very interested. [0]: https://github.com/QubitProducts/bamboo https://github.com/QubitProducts/bamboo

jolynch 11y ago
Check out the deep dive I wrote into how you can help prevent HAProxy reloads from dropping traffic without a large 99% hit over at http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html http://engineeringblog.yelp.com/2015/04/true-zero-downtime-h... Looks like bamboo encourages folks to use this strategy via a custom reload command: https://github.com/QubitProducts/bamboo/issues/152 https://github.com/QubitProducts/bamboo/issues/152
- fidget 11y ago
  That's egress traffic :) And yeah, https://github.com/QubitProducts/bamboo/issues/143#issuecomment-139306818 https://github.com/QubitProducts/bamboo/issues/143#issuecomm... is currently my best idea (with some additional nfq stuff). Doesn't help when requests span the entirety of the reload though.
  - jolynch 11y ago
    I mean, I wouldn't encourage anyone to use HAProxy for microservice load balancing unless they're running it on every node, at which point clients are always talking to localhost and everything is egress. That being said, you can turn ingress into egress with an ifb. A coworker of mine created a proof of concept for using our strategy with an external facing load balancer but I don't think he ever tried it in production so I'm not sure how well it works. To be fair I'm scared of both the iptables and tc solution on external LBs because you might never know that you're refusing connections accidentally. Linux 4.4 is coming with some patches that should help with this and afaik BSDs have done it right since the start.
TurningCanadian 11y ago
AFAIK Apache Traffic Server does zero-downtime reload http://trafficserver.readthedocs.org/en/latest/reference/configuration/remap.config.en.html http://trafficserver.readthedocs.org/en/latest/reference/con...
magiconair 11y ago
I'm curious whether https://github.com/eBay/fabio https://github.com/eBay/fabio would meet your needs. It picks up the routing table from consul changes and service registrations and reloads it without restart. All the services have to do is to register their routes in consul. It is meant as a replacement for consul-template+HAproxy. Disclosure: I'm the author
- fidget 11y ago
  Unfortunately consul integration isn't useful for us.
falcolas 11y ago
Live configuration via the stats socket is frequently overlooked, but works a treat. I'd recommend investigating it before giving up on haproxy.
- fidget 11y ago
  I use it post reload to persist the state of health checks, which I thought was pretty cool. However, I don't believe you can add a (new) server to a backend via the stats socket.
  - falcolas 11y ago
    > However, I don't believe you can add a (new) server to a backend via the stats socket. Yes, though a bit of advanced planning and creative configuration can usually work around this limitation. Making the assumption that new servers and services occur within a predictable range of IPs and ports (or have their own haproxy to route within the server), you can create a list of backend servers which are all disabled and just waiting to be individually enabled. I've done this in the past, and the disabled server entries never seem to have a negative impact on HAProxy. It also has the advantage that you can set your HAProxy instances to peer with each other, so the change only has to be made once to propagate to connected instances.

by fidget 11y ago

jolynch 11y ago

Check out the deep dive I wrote into how you can help prevent HAProxy reloads from dropping traffic without a large 99% hit over at http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html http://engineeringblog.yelp.com/2015/04/true-zero-downtime-h... Looks like bamboo encourages folks to use this strategy via a custom reload command: https://github.com/QubitProducts/bamboo/issues/152 https://github.com/QubitProducts/bamboo/issues/152

fidget 11y ago
That's egress traffic :) And yeah, https://github.com/QubitProducts/bamboo/issues/143#issuecomment-139306818 https://github.com/QubitProducts/bamboo/issues/143#issuecomm... is currently my best idea (with some additional nfq stuff). Doesn't help when requests span the entirety of the reload though.
- jolynch 11y ago
  I mean, I wouldn't encourage anyone to use HAProxy for microservice load balancing unless they're running it on every node, at which point clients are always talking to localhost and everything is egress. That being said, you can turn ingress into egress with an ifb. A coworker of mine created a proof of concept for using our strategy with an external facing load balancer but I don't think he ever tried it in production so I'm not sure how well it works. To be fair I'm scared of both the iptables and tc solution on external LBs because you might never know that you're refusing connections accidentally. Linux 4.4 is coming with some patches that should help with this and afaik BSDs have done it right since the start.

TurningCanadian 11y ago

AFAIK Apache Traffic Server does zero-downtime reload http://trafficserver.readthedocs.org/en/latest/reference/configuration/remap.config.en.html http://trafficserver.readthedocs.org/en/latest/reference/con...

magiconair 11y ago

I'm curious whether https://github.com/eBay/fabio https://github.com/eBay/fabio would meet your needs. It picks up the routing table from consul changes and service registrations and reloads it without restart. All the services have to do is to register their routes in consul. It is meant as a replacement for consul-template+HAproxy. Disclosure: I'm the author

fidget 11y ago
Unfortunately consul integration isn't useful for us.

falcolas 11y ago

Live configuration via the stats socket is frequently overlooked, but works a treat. I'd recommend investigating it before giving up on haproxy.

fidget 11y ago
I use it post reload to persist the state of health checks, which I thought was pretty cool. However, I don't believe you can add a (new) server to a backend via the stats socket.
- falcolas 11y ago
  > However, I don't believe you can add a (new) server to a backend via the stats socket. Yes, though a bit of advanced planning and creative configuration can usually work around this limitation. Making the assumption that new servers and services occur within a predictable range of IPs and ports (or have their own haproxy to route within the server), you can create a list of backend servers which are all disabled and just waiting to be individually enabled. I've done this in the past, and the disabled server entries never seem to have a negative impact on HAProxy. It also has the advantage that you can set your HAProxy instances to peer with each other, so the change only has to be made once to propagate to connected instances.