4 ms·

I wanted to like HAproxy but the lack of support for in-place zero downtime config reloads without resorting to witchcraft was a showstopper. Use case for us a

by jnsaff2 10y ago

I wanted to like HAproxy but the lack of support for in-place zero downtime config reloads without resorting to witchcraft was a showstopper. Use case for us a service discovery load balancer for Mesos/Marathon where frontend/backend config can change as often as every few seconds. Never had any issues with nginx.

kawsper 10y ago
Could you elaborate on how you use Nginx as a service discovery load balancer?
- jnsaff2 10y ago
  I think most of the ideas are presented here: https://github.com/martensson/nixy https://github.com/martensson/nixy In essence there is a wrapper that: - reads service information and current backend instances out of service catalog (either Marathon or Consul) - writes nginx configuration - runs nginx configtest - tells nginx to reload its configuration Nixy referenced above takes service information directly from Marathon event stream but I usually have Consul so I use homebaked consul-template + nginx solution.
user5994461 10y ago
>> where frontend/backend config can change as often as every few seconds That's a red flag. Don't make architectures where you depend on crazy requirements like that.
- jnsaff2 10y ago
  Ok, I will go tell the 200 or so developers that they can only deploy to test/staging once an hour or so instead of, like .. whenever they need. This is a PaaS supporting many micro-services. There is a hold-off to not reload more than once a second just as an insurance against runaway processes but beyond that nginx handles this without any sweat. Frequent reloads are currently in the non-prod cluster only but I don't see any reason why it would be an issue in prod as well.
  - user5994461 10y ago
    Depends what you handle. Some services really don't like being forcefully disconnected and not everything is a short lived HTTP(S) request. P.S. I am in finance, assume my requirements are severely more extreme than yours. Just like my errors.
    - jnsaff2 10y ago
      That's why we switched to nginx as it is perfectly fine reloading configuration without dropping existing connections. HAproxy can not do that. There is a black magic approach to having two HAproxies listening on the same port so there would always be something listening and connections could be drained, but it's dicey.

by jnsaff2 10y ago

kawsper 10y ago

Could you elaborate on how you use Nginx as a service discovery load balancer?

jnsaff2 10y ago
I think most of the ideas are presented here: https://github.com/martensson/nixy https://github.com/martensson/nixy In essence there is a wrapper that: - reads service information and current backend instances out of service catalog (either Marathon or Consul) - writes nginx configuration - runs nginx configtest - tells nginx to reload its configuration Nixy referenced above takes service information directly from Marathon event stream but I usually have Consul so I use homebaked consul-template + nginx solution.

user5994461 10y ago

>> where frontend/backend config can change as often as every few seconds That's a red flag. Don't make architectures where you depend on crazy requirements like that.

jnsaff2 10y ago
Ok, I will go tell the 200 or so developers that they can only deploy to test/staging once an hour or so instead of, like .. whenever they need. This is a PaaS supporting many micro-services. There is a hold-off to not reload more than once a second just as an insurance against runaway processes but beyond that nginx handles this without any sweat. Frequent reloads are currently in the non-prod cluster only but I don't see any reason why it would be an issue in prod as well.
- user5994461 10y ago
  Depends what you handle. Some services really don't like being forcefully disconnected and not everything is a short lived HTTP(S) request. P.S. I am in finance, assume my requirements are severely more extreme than yours. Just like my errors.
  - jnsaff2 10y ago
    That's why we switched to nginx as it is perfectly fine reloading configuration without dropping existing connections. HAproxy can not do that. There is a black magic approach to having two HAproxies listening on the same port so there would always be something listening and connections could be drained, but it's dicey.