3 ms·
The most significant issue with HAProxy and large SOAs is the lack of really good dynamic re-configuration. You don't want to have to manually change your confi
by jolynch 11y ago
The most significant issue with HAProxy and large SOAs is the lack of really good dynamic re-configuration. You don't want to have to manually change your configuration every time someone wants to add or change load balancing to a service, and if you do it automatically you run into HAProxy reloads being a reliability problem as they drop traffic on the floor due to Linux being goofy in how it implements SYN handling.
It looks like there is a lot of good advice on this thread for how to mitigate the configuration problem (bamboo, consul template, etc ...) but then you run into reloads, and even if you use zero downtime restarts as I described in my HAProxy deep dive (http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html http://engineeringblog.yelp.com/2015/04/true-zero-downtime-h...), you still end up with periodic ~50ms blips in your timings.
We've found that Synapse (https://github.com/airbnb/synapse https://github.com/airbnb/synapse) is best in class at managing HAProxy for us in a fairly large SOA. It is really intelligent about doing as much as possible dynamically over the HAProxy stats socket as opposed to reloading for every configuration change, and we can finely tune how often we are reloading HAProxy. The best part is that it is fully pluggable. Use DNS, marathon, zookeeper, etcd, etc...: Synapse can manifest that service registration information into HAProxy configurations.
- exelius 11y agoI personally prefer running a large number of logically separate HAProxy instances (a pair for each service). This makes it easier to forecast / tune request volume of the proxy servers on a per-service level, and it limits the impact that reconfiguration of one service can have on another (which helps immensely in root-cause analysis of production incidents once you've scaled beyond one team). It makes firewall rules a pain, but really no more painful than the general orchestration / configuration pain you feel at this scale. The good part about HAProxy is that it's flexible and can be configured any number of ways, but I've found that flexibility can weigh some teams by using it to solve problems that it's really not designed for. Sometimes it's better to use a brute-force approach with a simple configuration and deal with the complexity in a tool better designed for complexity like Mesos.
- fosk 11y agoOn a side note, Kong (https://github.com/Mashape/kong https://github.com/Mashape/kong), also mentioned in the article, allows re-configurations through its RESTful interface for every plugin.
- jrv 11y agoAnother problem with the configuration reloading is that all the backend/service metrics counters that HAProxy exposes on its metrics endpoint get reset on every reload. That makes for really noisy metrics gathering in situations where you have rapid successive changes to the config, such as gradually scaling up hundreds of instances of a new version of a microservice and then scaling down the old version. When you then scrape the metrics via Prometheus using an bridge like HAProxy Exporter (https://github.com/prometheus/haproxy_exporter https://github.com/prometheus/haproxy_exporter), you lose a lot of counter increments due to the frequent resets, so you get lower and noisy rates for that time period.
- jsmeaton 11y agoI hadn't heard of Prometheus before. We use nagios for alerting/alarms, and graphite/grafana separately for metrics. Would you recommend looking into prometheus as an alternative?
- jrv 11y agoAs a Prometheus author, I'm biased, but yes :) See also http://prometheus.io/ http://prometheus.io/ and https://www.youtube.com/watch?v=HjN23GgCzQY https://www.youtube.com/watch?v=HjN23GgCzQY for an intro.