9 ms·

HAProxy 2.4

159 points by throwaway888abc 5y ago

smitty1e 5y ago
If I have a cloud provider, why might I use HAProxy over the provider's load balancing solution?
- toast0 5y ago
  More control, maybe less cost? If your cloudy provider has an awful solution like the one I was using, you'd get better uptime by doing it yourself. I haven't heard many problems with AWS or Google load balancing, but sometimes there's issues where people talk about 'pre-warming'... more control would probably help there. If you're running HAProxy on AWS, be sure your AWS firewall rules are stateless, otherwise you may well run into the unpublished connection tracking limits.
  - vladvasiliu 5y ago
    > If you're running HAProxy on AWS, be sure your AWS firewall rules are stateless, otherwise you may well run into the unpublished connection tracking limits. This. I got bit by this the other day, a bunch of servers were having very low CPU usage but new connections to them were failing. Turns out it was because they were hitting the conntrack limit. Changing the security group rules to allow all from all "fixed" it. Although after a quick test, I think there's another limit, also unpublished, as I still can't establish as many connections as I want on a smaller instance – same number of requests on fewer connections works, and also a bigger instance works with many connections. In every case, the CPU was largely unused. AWS doc regarding the issue: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/security-group-connection-tracking.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/security...
    - toast0 5y ago
      > Although after a quick test, I think there's another limit, also unpublished, as I still can't establish as many connections as I want on a smaller instance. Any chance this is an OS limit in your instance? Either a connection tracking firewall there, or FD/socket count limits? I don't have experience beyond the AWS connection tracking limits because I ran into them while testing something, but only found out how to avoid them much later (the test didn't go well, because of the limits, so we didn't go forward, and our assigned rep didn't tell us about the workaround either; I only found out by complaining about the limit to enough people that someone knew how to fix it)
      - vladvasiliu 5y ago
        I doubt that, because the same exact VM, after changing to a beefier instance type (c5.4xl, instead of the original t3.micro and later c5.large) accepted the connections without any problem. If it's OS related, it may be some dynamic limit based on RAM size. But the limit was something around 100-200 connections, so nothing particularly crazy. The way I found out about the limit was that downstream servers couldn't establish connections, and a manual test showed significant packet loss, while the servers were under 10% CPU and accepted connections locally. I opened a support ticket and the guy apparently saw that they were hitting the conntrack limit. Once I replaced the security group with a blanket allow all as he suggested, connections could be established again. But two days later, when the number of connections increased further (while the security group was still allow all), I started seeing again connections not being established, although the servers were still not particularly loaded. As these are UDP (DNS) servers, I can't usefully quantify the number of different clients. I set up a quick lab with two isolated EC2 instances and figured there must be another limit to the number of connections, although it seems somewhat higher – or maybe the conntrack limit as shown by ethtool doesn't update immediately. This happened yesterday, Saturday, while I was on call, so I didn't push the tests far enough to reach any clear-cut conclusion. I just doubled the servers and everything went back to normal. However, the limit does seem to be related to the instance size and per interface. Meaning that adding another interface doubled the connections I could establish. Again, I didn't push the tests any further, though I intend to this sometime next week.
- jen20 5y ago
  One potential use case: proxying targets with no fixed IP addresses for services which require fixed IP addresses. For example, exposing RDS databases via privatelink. It would be nice if AWS would address this natively, but HAProxy will do.
- paulddraper 5y ago
  You'd have more specific than "every single cloud provider" but in general HAProxy is way more advanced than most other proxies. Stickiness, rate limiting, gzip compression, configurable logging, caching, Lua scripting, load balancing to domain names, more options for health lifecycle. It even has a control socket to instantly manage the server without restart. HAProxy is actually the inventor the PROXY protocol which a number of other systems have now copied. The convenience of cloud proxies is no small matter, but on a feature-by-feature basis HAProxy easily dominates.
  - rogerdonut 5y ago
    Well said, to add to this, HAProxy is also closing the gap on convenience with its recent addition of AWS EC2 service discovery [1] and Consul service discovery integrations [2]. [1] https://www.haproxy.com/blog/announcing-haproxy-data-plane-api-2-3/#aws-ec2-service-discovery https://www.haproxy.com/blog/announcing-haproxy-data-plane-a... [2] https://www.haproxy.com/blog/announcing-haproxy-data-plane-api-2-2/#consul-service-discovery https://www.haproxy.com/blog/announcing-haproxy-data-plane-a...
    - orthecreedence 5y ago
      Wow, cool. HAProxy is coming a long way. Consul integration makes me excited to take another look at using it for our mesh. Using Fabio right now, and it's fine, but I'd love to go back to HAProxy for this if it makes sense.
- rogerdonut 5y ago
  disclaimer: I work at HAProxy Technologies More control/flexibility (i.e. cloud provider usually gives you a limited number of "checkboxes" that they want you to be able to modify), generally better observability (HAProxy timing metrics [1] and session state at disconnection codes [2] are very helpful for debugging), and granular control over timeouts [3]. You also get builtin rate limiting [4], which from what I have seen with most cloud load balancers usually requires an up-charge through their "WAF" product. Better health checking, circuit breaking, and layer4/layer7 retries [5]. Another consideration is whether you have spiky traffic patterns. For AWS load balancers at least, they need time to "warm up" to larger traffic levels. If you manage your own load balancing tier you can scale on demand. [1] https://www.haproxy.com/documentation/hapee/latest/onepage/#8.4 https://www.haproxy.com/documentation/hapee/latest/onepage/#... [2] https://www.haproxy.com/documentation/hapee/latest/onepage/#8.5 https://www.haproxy.com/documentation/hapee/latest/onepage/#... [3] https://www.haproxy.com/documentation/hapee/latest/onepage/#4.2-timeout%20client https://www.haproxy.com/documentation/hapee/latest/onepage/#... [4] https://www.haproxy.com/blog/four-examples-of-haproxy-rate-limiting/ https://www.haproxy.com/blog/four-examples-of-haproxy-rate-l... [5] https://www.haproxy.com/blog/haproxy-layer-7-retries-and-chaos-engineering/ https://www.haproxy.com/blog/haproxy-layer-7-retries-and-cha...
  - aidos 5y ago
    I last used HAProxy about a decade ago for fairly simple load balancing and caching - worked really well. Question about the suitability for a dynamic backend case. I have a system where clients are assigned to backends exclusively for the duration of a session. At the moment I have a pool of backends and each client is explicitly told which backend to use so they can route to it. Ideally I’d make this transparent so it was easier to reassign on the fly. Is this case reasonably supported with HAProxy and which features should I be looking at to get a sense of it? I’m working under the assumption that I’ll need to build the machinery to maintain the pool and registry of where clients should be routed to.
    - rogerdonut 5y ago
      While I have not tested this specific use case, I would start by exploring map files [1] and/or stick tables [2]. Map files can be updated on-the-fly through the HAProxy Runtime API. With stick tables you can persist a session/client based on anything found within the request headers, which would include a users session id. [1] https://www.haproxy.com/blog/introduction-to-haproxy-maps/ https://www.haproxy.com/blog/introduction-to-haproxy-maps/ [2] https://www.haproxy.com/blog/introduction-to-haproxy-stick-tables/ https://www.haproxy.com/blog/introduction-to-haproxy-stick-t...
- OhSoHumble 5y ago
  Cost. I imagine.
  - mnahkies 5y ago
    To expand on this I believe an aws load balancer is $30 / mo. For early stage projects, and particularly for personal projects it's more cost efficient to use haproxy. As an added bonus it's got all the bells and whistles to match demands as you scale, and no (cloud) vendor lock-in
- topspin 5y ago
  Missing capabilities. Some cloud provider load balancers don't support mutual authentication with TLS, for example.
- kureikain 5y ago
  They have very nice features that provider's load balancing might not offer you. First to me is the ability to retain the IP address. When using AWS Load Balancer, the LB scale automatically and replace the IP. Second is sophisiciated routing. Example route by cookie. What it buy is that by setting a cookie you can make sure the request hit a specific pool. You can also have concept of backup Third is lots of useful metrics with their stats page. Fourth is the ability to alter the responses such as removing header. You have to pair with lambda edge for these on AWS. What it helps is imagine you can write back the user_id to HAProxy, HAProxy log it together with the request, and delete it(so client won't see it). It helps to easily tail the log for a specific user. And a lot of rate limiting built-in features that just take a few lines of config.
  - ncsurfus 5y ago
    Small note - AWS Network Load Balancers maintain the same IP addresses.
    - harpratap 5y ago
      Yeah I was confused too. Why on earth will someone provide a load balancer if it keeps changing it's IP address as it scales?
      - saurik 5y ago
        The normal AWS load balancers are set up to stay in service as long as the cache time on their IP address having been served by DNS... (which, FWIW, I have long argued "isn't sufficient", as many ISPs in countries half a world away from the US or using tech like satellites don't honor DNS cache times and then your requests end up getting routed to someone else... I routinely got tons of HTTP requests clearly destined for someone else's product).
      - ris 5y ago
        Because they're intended to be referenced via hostname, and yes, regular HTTP AWS load balancers do indeed rotate their IPs.
- dilyevsky 5y ago
  To add to what other comments have already mentioned - cloud lbs send you random resets. If you don’t want to have your tcp sessions fail randomly under normal operation you have to use your own load balancer
- thayne 5y ago
  We actually use both. We have the cloud provider's load balancers balancing to HAProxy load balancers. We use HAProxy, because it is much more flexible and powerful the the cloud provider's LB, and we use the cloud provider LB because it integrates because it is simpler to terminate public TLS with them and manage the certificates.
- NDizzle 5y ago
  Looking at the standard haproxy error pages present when using an Azure App Service, you might be surprised to learn that the cloud providers are using haproxy for you, behind the scenes.
- gregmac 5y ago
  We use it at work for hosting an app designed pre-cloud (using one db per tenant) in AWS: We deploy auto-scaling groups for each major app version (because customers can choose when to upgrade, so we can have anywhere from 1-3 versions live at a time). There's a database that has an entry for each customer system, with one or more domains, the app version and the db name (usually auto-generated but for historical reasons can be set manually). There's a UI to manage all this. A script takes this data and builds haproxy config, creating backends for each version group and routing rules for domain to the proper group. This part could maybe be done with ALB now, but I am not certain of that. We also automatically configure SSL for all domains: anything that doesn't have a static .pem file gets LetsEncrypt cert. Most of these are done via HTTP-01 because they're customer-owned domains that just CNANE to us. None of this is doable via AWS built-in stuff. There's also a bunch of other hacks that haproxy does: doesn't redirect to SSL for a couple specific (non sensitive) URLs+user agent that doesn't follow redirects; returns a fake "success" page for a long-gone service called by an obsolete client some customers of (ex-)customers are still running, which effective causes a DDoS attack due to retries if we return 404 or 5xx; Has some awareness of backend state and shows better error pages than just a generic 5xx depending on situation. Haproxy instances are behind NLB, but otherwise there's a single (layer 7) hop to the app server. The end result is you can configure a new system via our management UI, and so long as DNS is setup (using a wildcard subdomain we own, and/or customer's CNAME entry) within a few minutes it will be live (database deployed, app servers aware of connection/domain mapping, proxy configured with SSL).
rogerdonut 5y ago
Author of the blog post here, happy to answer any questions about this release.
- thegagne 5y ago
  Open tracing sounds nice, what info is passed along?
  - paulddraper 5y ago
    My question is what conversation has there been re: Opentelemetry, the successor to OT
    - bostonsre 5y ago
      It sounds like ot is still in beta (but it's a little confusing that they're 1.0 but it has a message saying they're in beta on their main page). It looks like it's being adopted by a few larger app monitoring products, wondering if datadog will follow suit or if they will stick with open tracing and their custom implementations for logging and metrics. I assume they will support ingestion from ot at some point though. It sounds neat tho, will definitely read up on it more.
  - rogerdonut 5y ago
    You can pass along timing data from every aspect of the request [1] currently there are approximately 20 events supported. You can also send over just about anything that can be stored within an HAProxy variable / header through a tag. Here's an example configuration for HAProxy (simple) [2] (advanced) [3] and the OpenTracing specific configuration (simple) [4] (advanced) [5]. I've included both simple and advanced configurations. [1] https://github.com/haproxy/haproxy/blob/master/addons/ot/README#L410 https://github.com/haproxy/haproxy/blob/master/addons/ot/REA... [2] https://github.com/haproxy/haproxy/blob/master/addons/ot/test/be/haproxy.cfg https://github.com/haproxy/haproxy/blob/master/addons/ot/tes... [3] https://github.com/haproxy/haproxy/blob/master/addons/ot/test/ctx/haproxy.cfg https://github.com/haproxy/haproxy/blob/master/addons/ot/tes... [4] https://github.com/haproxy/haproxy/blob/master/addons/ot/test/be/ot.cfg https://github.com/haproxy/haproxy/blob/master/addons/ot/tes... [5] https://github.com/haproxy/haproxy/blob/master/addons/ot/test/ctx/ot.cfg https://github.com/haproxy/haproxy/blob/master/addons/ot/tes...
pull_my_finger 5y ago
I've been long interested in HAProxy, but I can't seem to sell myself on it. When might I choose HAProxy that OpenResty[1] would not be sufficient or better? [1]: https://openresty.org/en/ https://openresty.org/en/
- rogerdonut 5y ago
  IMO you would use OpenResty as a web server and HAProxy as a load balancer in front of it. HAProxy focuses on being a high-performance load balancer. OpenResty is built ontop of NGINX and many of the load balancing features in NGINX are locked behind their "Plus" product, all of which are provided by default within HAProxy. In order to take advantage of these features in OpenResty (such as active health checking and DNS service discovery) you would need to use Lua scripts where in HAProxy they are implemented in the core.
- ignoramous 5y ago
  HAProxy as a load balancer / load shedder / traffic shaper is what I've seen it used for.
- TimWolla 5y ago
  Disclosure: Community contributor to HAProxy, I help maintain HAProxy's issue tracker. HAProxy is virtually unusable as a file server, but really shines as a reverse proxy. One of the main reasons I almost always deploy HAProxy in tandem with (stock) nginx is the superior HTTP rewriting / configuration. See also my comment back on HAProxy 2.0: https://news.ycombinator.com/item?id=20198232 https://news.ycombinator.com/item?id=20198232 HAProxy's configuration is a procedural style configuration. The HTTP rules (e.g. adding or deleting headers) are processed in order if their condition matches. (Stock) nginx has a declarative configuration which makes it hard for me to understand which options apply when and the inheritance by nesting different blocks is sometimes very confusing. One example would be an `add_header` in a `location` completely overriding (instead of extending) an `add_header` in a `server`. So if I want to put e.g. HSTS into the server to apply to the full host and also want to add headers for different paths, then I need to duplicate the HSTS stuff.
bullen 5y ago
HAProxy is great software, mostly because Willy can focus. But I feel sorry for him now, with HTTP/2 and HTTP/3 the pain will never stop. And IPv6 makes most of this obsolete if it happens! I stick with HTTP/1.1 since I'm building a native 3D MMO; only Microsoft can block port 80 which I doubt they will atleast in the coming 5 years! Linux wont, they are not that stupid. Edit: allready the bots have downvoted, this is beautiful.
- AlexSW 5y ago
  I can't tell whether I'm missing something: how would IPv6 obsolete their work?
  - bullen 5y ago
    The only "real" reason you need a loadbalancer is that you want to save IP addresses. If all your machines have their own unique public address you can loadbalance with your own software by pointing people directly to different machines.
    - racingmars 5y ago
      > The only "real" reason you need a loadbalancer is that you want to save IP addresses. What?! Load balancers tend to be in place for availability and scalability reasons, not to save IP addresses. Sure, there are many use cases where you might not need to have public IPs on the backends a load balancer is sending traffic to, but to say that's the only "real" reason for a load balancer seems to be missing the point.
    - purerandomness 5y ago
      A load balancer is, for example, used to distribute traffic to multiple worker nodes to scale horizontally. It can also be used for canary deployments where you roll out a feature to 1% of your user base. There are several other use cases. I'm pretty sure 'saving IP addresses' is not a use case you'd use a load balancer for.
    - zbentley 5y ago
snvzz 5y ago
The .org[0] continues to be the better site. [0] https://www.haproxy.org/ https://www.haproxy.org/
kzrdude 5y ago
Any blog post or release announcement that is expected to get wide dissemination should preferably have a blurb that explains what the project/product is, so that the post is accessible to more people. Maybe something like "The Reliable, High Performance TCP/HTTP Load Balancer".
- osrec 5y ago
  HAProxy is so well known that it really doesn't need to. I mean, you wouldn't expect Facebook to mention the fact that it's a social network on every one of its blog posts. And if you don't know of the product, just Google it.
  - AdrianB1 5y ago
    It is a matter of choice. Do you mind adding a sentence for the people that are too lazy to do some search? Maybe if they are too lazy they are not interested in the product, but the cost of that sentence is negligible. Even Facebook was a lot less popular at some time and my mother asked me what is that.
    - osrec 5y ago
      The cost of the sentence is screen real estate, plus a distraction from the purpose of the post. I sometimes feel the collective of spirit of HN can never be satisfied. If you do put that sentence in, someone will probably question why it's there :p
- nickramirez 5y ago
  kzrdude, great feedback. These big version announcements gather the attention of people who may not be familiar with HAProxy and so, I'd say they are a unique case where we should introduce the product for newcomers.
osrec 5y ago
I can only imagine some investment banks have donated a fair bit given that HAProxy now has built in support for FIX. I really wasn't expecting to see that!
- deleted 5y ago
  [deleted]
- oogali 5y ago
  Since ~2004, folks in the financial industry have been migrating away from FIX to other lighter protocols such as ITCH, OUCH, and SBE (standardized from FAST: "FIX Adapted for Streaming"). This is analogous to the move we're all witnessing from HTTP to HTTP/2 (standardized from SPDY) and HTTP/3 (QUIC). The desire to reduce the amount of bandwidth spent pushing uncompressed text around and reduce the amount of CPU needed to parse strings is what spurred market participants to explore alternatives to FIX, namely in the form of binary encoded message that costs less to parse (from a CPU and bandwidth perspective). As libraries release support for alternative protocols and the industry compiles their projects against these newer versions, the adoption of non-FIX will keep increasing. Similar to how you get HTTP/2 for "free" when using the Go stdlib so its downstream consumers such as Caddy and Traefik can provide that as a default to end users. https://en.wikipedia.org/wiki/List_of_electronic_trading_protocols https://en.wikipedia.org/wiki/List_of_electronic_trading_pro... EDIT: Added year for context on timeframe of first implementations
carlivar 5y ago
We love haproxy. It is the load balancer powering millions of requests per second in aggregate in our data centers. These data centers cost less than half what the cloud would.

HAProxy 2.4

smitty1e 5y ago

If I have a cloud provider, why might I use HAProxy over the provider's load balancing solution?

toast0 5y ago
More control, maybe less cost? If your cloudy provider has an awful solution like the one I was using, you'd get better uptime by doing it yourself. I haven't heard many problems with AWS or Google load balancing, but sometimes there's issues where people talk about 'pre-warming'... more control would probably help there. If you're running HAProxy on AWS, be sure your AWS firewall rules are stateless, otherwise you may well run into the unpublished connection tracking limits.
- vladvasiliu 5y ago
  > If you're running HAProxy on AWS, be sure your AWS firewall rules are stateless, otherwise you may well run into the unpublished connection tracking limits. This. I got bit by this the other day, a bunch of servers were having very low CPU usage but new connections to them were failing. Turns out it was because they were hitting the conntrack limit. Changing the security group rules to allow all from all "fixed" it. Although after a quick test, I think there's another limit, also unpublished, as I still can't establish as many connections as I want on a smaller instance – same number of requests on fewer connections works, and also a bigger instance works with many connections. In every case, the CPU was largely unused. AWS doc regarding the issue: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/security-group-connection-tracking.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/security...
  - toast0 5y ago
    > Although after a quick test, I think there's another limit, also unpublished, as I still can't establish as many connections as I want on a smaller instance. Any chance this is an OS limit in your instance? Either a connection tracking firewall there, or FD/socket count limits? I don't have experience beyond the AWS connection tracking limits because I ran into them while testing something, but only found out how to avoid them much later (the test didn't go well, because of the limits, so we didn't go forward, and our assigned rep didn't tell us about the workaround either; I only found out by complaining about the limit to enough people that someone knew how to fix it)
    - vladvasiliu 5y ago
      I doubt that, because the same exact VM, after changing to a beefier instance type (c5.4xl, instead of the original t3.micro and later c5.large) accepted the connections without any problem. If it's OS related, it may be some dynamic limit based on RAM size. But the limit was something around 100-200 connections, so nothing particularly crazy. The way I found out about the limit was that downstream servers couldn't establish connections, and a manual test showed significant packet loss, while the servers were under 10% CPU and accepted connections locally. I opened a support ticket and the guy apparently saw that they were hitting the conntrack limit. Once I replaced the security group with a blanket allow all as he suggested, connections could be established again. But two days later, when the number of connections increased further (while the security group was still allow all), I started seeing again connections not being established, although the servers were still not particularly loaded. As these are UDP (DNS) servers, I can't usefully quantify the number of different clients. I set up a quick lab with two isolated EC2 instances and figured there must be another limit to the number of connections, although it seems somewhat higher – or maybe the conntrack limit as shown by ethtool doesn't update immediately. This happened yesterday, Saturday, while I was on call, so I didn't push the tests far enough to reach any clear-cut conclusion. I just doubled the servers and everything went back to normal. However, the limit does seem to be related to the instance size and per interface. Meaning that adding another interface doubled the connections I could establish. Again, I didn't push the tests any further, though I intend to this sometime next week.
jen20 5y ago
One potential use case: proxying targets with no fixed IP addresses for services which require fixed IP addresses. For example, exposing RDS databases via privatelink. It would be nice if AWS would address this natively, but HAProxy will do.
paulddraper 5y ago
You'd have more specific than "every single cloud provider" but in general HAProxy is way more advanced than most other proxies. Stickiness, rate limiting, gzip compression, configurable logging, caching, Lua scripting, load balancing to domain names, more options for health lifecycle. It even has a control socket to instantly manage the server without restart. HAProxy is actually the inventor the PROXY protocol which a number of other systems have now copied. The convenience of cloud proxies is no small matter, but on a feature-by-feature basis HAProxy easily dominates.
- rogerdonut 5y ago
  Well said, to add to this, HAProxy is also closing the gap on convenience with its recent addition of AWS EC2 service discovery [1] and Consul service discovery integrations [2]. [1] https://www.haproxy.com/blog/announcing-haproxy-data-plane-api-2-3/#aws-ec2-service-discovery https://www.haproxy.com/blog/announcing-haproxy-data-plane-a... [2] https://www.haproxy.com/blog/announcing-haproxy-data-plane-api-2-2/#consul-service-discovery https://www.haproxy.com/blog/announcing-haproxy-data-plane-a...
  - orthecreedence 5y ago
    Wow, cool. HAProxy is coming a long way. Consul integration makes me excited to take another look at using it for our mesh. Using Fabio right now, and it's fine, but I'd love to go back to HAProxy for this if it makes sense.
rogerdonut 5y ago
disclaimer: I work at HAProxy Technologies More control/flexibility (i.e. cloud provider usually gives you a limited number of "checkboxes" that they want you to be able to modify), generally better observability (HAProxy timing metrics [1] and session state at disconnection codes [2] are very helpful for debugging), and granular control over timeouts [3]. You also get builtin rate limiting [4], which from what I have seen with most cloud load balancers usually requires an up-charge through their "WAF" product. Better health checking, circuit breaking, and layer4/layer7 retries [5]. Another consideration is whether you have spiky traffic patterns. For AWS load balancers at least, they need time to "warm up" to larger traffic levels. If you manage your own load balancing tier you can scale on demand. [1] https://www.haproxy.com/documentation/hapee/latest/onepage/#8.4 https://www.haproxy.com/documentation/hapee/latest/onepage/#... [2] https://www.haproxy.com/documentation/hapee/latest/onepage/#8.5 https://www.haproxy.com/documentation/hapee/latest/onepage/#... [3] https://www.haproxy.com/documentation/hapee/latest/onepage/#4.2-timeout%20client https://www.haproxy.com/documentation/hapee/latest/onepage/#... [4] https://www.haproxy.com/blog/four-examples-of-haproxy-rate-limiting/ https://www.haproxy.com/blog/four-examples-of-haproxy-rate-l... [5] https://www.haproxy.com/blog/haproxy-layer-7-retries-and-chaos-engineering/ https://www.haproxy.com/blog/haproxy-layer-7-retries-and-cha...
- aidos 5y ago
  I last used HAProxy about a decade ago for fairly simple load balancing and caching - worked really well. Question about the suitability for a dynamic backend case. I have a system where clients are assigned to backends exclusively for the duration of a session. At the moment I have a pool of backends and each client is explicitly told which backend to use so they can route to it. Ideally I’d make this transparent so it was easier to reassign on the fly. Is this case reasonably supported with HAProxy and which features should I be looking at to get a sense of it? I’m working under the assumption that I’ll need to build the machinery to maintain the pool and registry of where clients should be routed to.
  - rogerdonut 5y ago
    While I have not tested this specific use case, I would start by exploring map files [1] and/or stick tables [2]. Map files can be updated on-the-fly through the HAProxy Runtime API. With stick tables you can persist a session/client based on anything found within the request headers, which would include a users session id. [1] https://www.haproxy.com/blog/introduction-to-haproxy-maps/ https://www.haproxy.com/blog/introduction-to-haproxy-maps/ [2] https://www.haproxy.com/blog/introduction-to-haproxy-stick-tables/ https://www.haproxy.com/blog/introduction-to-haproxy-stick-t...
OhSoHumble 5y ago
Cost. I imagine.
- mnahkies 5y ago
  To expand on this I believe an aws load balancer is $30 / mo. For early stage projects, and particularly for personal projects it's more cost efficient to use haproxy. As an added bonus it's got all the bells and whistles to match demands as you scale, and no (cloud) vendor lock-in
topspin 5y ago
Missing capabilities. Some cloud provider load balancers don't support mutual authentication with TLS, for example.
kureikain 5y ago
They have very nice features that provider's load balancing might not offer you. First to me is the ability to retain the IP address. When using AWS Load Balancer, the LB scale automatically and replace the IP. Second is sophisiciated routing. Example route by cookie. What it buy is that by setting a cookie you can make sure the request hit a specific pool. You can also have concept of backup Third is lots of useful metrics with their stats page. Fourth is the ability to alter the responses such as removing header. You have to pair with lambda edge for these on AWS. What it helps is imagine you can write back the user_id to HAProxy, HAProxy log it together with the request, and delete it(so client won't see it). It helps to easily tail the log for a specific user. And a lot of rate limiting built-in features that just take a few lines of config.
- ncsurfus 5y ago
  Small note - AWS Network Load Balancers maintain the same IP addresses.
  - harpratap 5y ago
    Yeah I was confused too. Why on earth will someone provide a load balancer if it keeps changing it's IP address as it scales?
    - saurik 5y ago
      The normal AWS load balancers are set up to stay in service as long as the cache time on their IP address having been served by DNS... (which, FWIW, I have long argued "isn't sufficient", as many ISPs in countries half a world away from the US or using tech like satellites don't honor DNS cache times and then your requests end up getting routed to someone else... I routinely got tons of HTTP requests clearly destined for someone else's product).
    - ris 5y ago
      Because they're intended to be referenced via hostname, and yes, regular HTTP AWS load balancers do indeed rotate their IPs.
dilyevsky 5y ago
To add to what other comments have already mentioned - cloud lbs send you random resets. If you don’t want to have your tcp sessions fail randomly under normal operation you have to use your own load balancer
thayne 5y ago
We actually use both. We have the cloud provider's load balancers balancing to HAProxy load balancers. We use HAProxy, because it is much more flexible and powerful the the cloud provider's LB, and we use the cloud provider LB because it integrates because it is simpler to terminate public TLS with them and manage the certificates.
NDizzle 5y ago
Looking at the standard haproxy error pages present when using an Azure App Service, you might be surprised to learn that the cloud providers are using haproxy for you, behind the scenes.
gregmac 5y ago
We use it at work for hosting an app designed pre-cloud (using one db per tenant) in AWS: We deploy auto-scaling groups for each major app version (because customers can choose when to upgrade, so we can have anywhere from 1-3 versions live at a time). There's a database that has an entry for each customer system, with one or more domains, the app version and the db name (usually auto-generated but for historical reasons can be set manually). There's a UI to manage all this. A script takes this data and builds haproxy config, creating backends for each version group and routing rules for domain to the proper group. This part could maybe be done with ALB now, but I am not certain of that. We also automatically configure SSL for all domains: anything that doesn't have a static .pem file gets LetsEncrypt cert. Most of these are done via HTTP-01 because they're customer-owned domains that just CNANE to us. None of this is doable via AWS built-in stuff. There's also a bunch of other hacks that haproxy does: doesn't redirect to SSL for a couple specific (non sensitive) URLs+user agent that doesn't follow redirects; returns a fake "success" page for a long-gone service called by an obsolete client some customers of (ex-)customers are still running, which effective causes a DDoS attack due to retries if we return 404 or 5xx; Has some awareness of backend state and shows better error pages than just a generic 5xx depending on situation. Haproxy instances are behind NLB, but otherwise there's a single (layer 7) hop to the app server. The end result is you can configure a new system via our management UI, and so long as DNS is setup (using a wildcard subdomain we own, and/or customer's CNAME entry) within a few minutes it will be live (database deployed, app servers aware of connection/domain mapping, proxy configured with SSL).

rogerdonut 5y ago

Author of the blog post here, happy to answer any questions about this release.

thegagne 5y ago
Open tracing sounds nice, what info is passed along?
- paulddraper 5y ago
  My question is what conversation has there been re: Opentelemetry, the successor to OT
  - bostonsre 5y ago
    It sounds like ot is still in beta (but it's a little confusing that they're 1.0 but it has a message saying they're in beta on their main page). It looks like it's being adopted by a few larger app monitoring products, wondering if datadog will follow suit or if they will stick with open tracing and their custom implementations for logging and metrics. I assume they will support ingestion from ot at some point though. It sounds neat tho, will definitely read up on it more.
- rogerdonut 5y ago
  You can pass along timing data from every aspect of the request [1] currently there are approximately 20 events supported. You can also send over just about anything that can be stored within an HAProxy variable / header through a tag. Here's an example configuration for HAProxy (simple) [2] (advanced) [3] and the OpenTracing specific configuration (simple) [4] (advanced) [5]. I've included both simple and advanced configurations. [1] https://github.com/haproxy/haproxy/blob/master/addons/ot/README#L410 https://github.com/haproxy/haproxy/blob/master/addons/ot/REA... [2] https://github.com/haproxy/haproxy/blob/master/addons/ot/test/be/haproxy.cfg https://github.com/haproxy/haproxy/blob/master/addons/ot/tes... [3] https://github.com/haproxy/haproxy/blob/master/addons/ot/test/ctx/haproxy.cfg https://github.com/haproxy/haproxy/blob/master/addons/ot/tes... [4] https://github.com/haproxy/haproxy/blob/master/addons/ot/test/be/ot.cfg https://github.com/haproxy/haproxy/blob/master/addons/ot/tes... [5] https://github.com/haproxy/haproxy/blob/master/addons/ot/test/ctx/ot.cfg https://github.com/haproxy/haproxy/blob/master/addons/ot/tes...

pull_my_finger 5y ago

I've been long interested in HAProxy, but I can't seem to sell myself on it. When might I choose HAProxy that OpenResty[1] would not be sufficient or better? [1]: https://openresty.org/en/ https://openresty.org/en/

rogerdonut 5y ago
IMO you would use OpenResty as a web server and HAProxy as a load balancer in front of it. HAProxy focuses on being a high-performance load balancer. OpenResty is built ontop of NGINX and many of the load balancing features in NGINX are locked behind their "Plus" product, all of which are provided by default within HAProxy. In order to take advantage of these features in OpenResty (such as active health checking and DNS service discovery) you would need to use Lua scripts where in HAProxy they are implemented in the core.
ignoramous 5y ago
HAProxy as a load balancer / load shedder / traffic shaper is what I've seen it used for.
TimWolla 5y ago
Disclosure: Community contributor to HAProxy, I help maintain HAProxy's issue tracker. HAProxy is virtually unusable as a file server, but really shines as a reverse proxy. One of the main reasons I almost always deploy HAProxy in tandem with (stock) nginx is the superior HTTP rewriting / configuration. See also my comment back on HAProxy 2.0: https://news.ycombinator.com/item?id=20198232 https://news.ycombinator.com/item?id=20198232 HAProxy's configuration is a procedural style configuration. The HTTP rules (e.g. adding or deleting headers) are processed in order if their condition matches. (Stock) nginx has a declarative configuration which makes it hard for me to understand which options apply when and the inheritance by nesting different blocks is sometimes very confusing. One example would be an `add_header` in a `location` completely overriding (instead of extending) an `add_header` in a `server`. So if I want to put e.g. HSTS into the server to apply to the full host and also want to add headers for different paths, then I need to duplicate the HSTS stuff.

bullen 5y ago

HAProxy is great software, mostly because Willy can focus. But I feel sorry for him now, with HTTP/2 and HTTP/3 the pain will never stop. And IPv6 makes most of this obsolete if it happens! I stick with HTTP/1.1 since I'm building a native 3D MMO; only Microsoft can block port 80 which I doubt they will atleast in the coming 5 years! Linux wont, they are not that stupid. Edit: allready the bots have downvoted, this is beautiful.

AlexSW 5y ago
I can't tell whether I'm missing something: how would IPv6 obsolete their work?
- bullen 5y ago
  The only "real" reason you need a loadbalancer is that you want to save IP addresses. If all your machines have their own unique public address you can loadbalance with your own software by pointing people directly to different machines.
  - racingmars 5y ago
    > The only "real" reason you need a loadbalancer is that you want to save IP addresses. What?! Load balancers tend to be in place for availability and scalability reasons, not to save IP addresses. Sure, there are many use cases where you might not need to have public IPs on the backends a load balancer is sending traffic to, but to say that's the only "real" reason for a load balancer seems to be missing the point.
  - purerandomness 5y ago
    A load balancer is, for example, used to distribute traffic to multiple worker nodes to scale horizontally. It can also be used for canary deployments where you roll out a feature to 1% of your user base. There are several other use cases. I'm pretty sure 'saving IP addresses' is not a use case you'd use a load balancer for.
  - zbentley 5y ago

snvzz 5y ago

The .org[0] continues to be the better site. [0] https://www.haproxy.org/ https://www.haproxy.org/

kzrdude 5y ago

Any blog post or release announcement that is expected to get wide dissemination should preferably have a blurb that explains what the project/product is, so that the post is accessible to more people. Maybe something like "The Reliable, High Performance TCP/HTTP Load Balancer".

osrec 5y ago
HAProxy is so well known that it really doesn't need to. I mean, you wouldn't expect Facebook to mention the fact that it's a social network on every one of its blog posts. And if you don't know of the product, just Google it.
- AdrianB1 5y ago
  It is a matter of choice. Do you mind adding a sentence for the people that are too lazy to do some search? Maybe if they are too lazy they are not interested in the product, but the cost of that sentence is negligible. Even Facebook was a lot less popular at some time and my mother asked me what is that.
  - osrec 5y ago
    The cost of the sentence is screen real estate, plus a distraction from the purpose of the post. I sometimes feel the collective of spirit of HN can never be satisfied. If you do put that sentence in, someone will probably question why it's there :p
nickramirez 5y ago
kzrdude, great feedback. These big version announcements gather the attention of people who may not be familiar with HAProxy and so, I'd say they are a unique case where we should introduce the product for newcomers.

osrec 5y ago

I can only imagine some investment banks have donated a fair bit given that HAProxy now has built in support for FIX. I really wasn't expecting to see that!

deleted 5y ago
[deleted]
oogali 5y ago
Since ~2004, folks in the financial industry have been migrating away from FIX to other lighter protocols such as ITCH, OUCH, and SBE (standardized from FAST: "FIX Adapted for Streaming"). This is analogous to the move we're all witnessing from HTTP to HTTP/2 (standardized from SPDY) and HTTP/3 (QUIC). The desire to reduce the amount of bandwidth spent pushing uncompressed text around and reduce the amount of CPU needed to parse strings is what spurred market participants to explore alternatives to FIX, namely in the form of binary encoded message that costs less to parse (from a CPU and bandwidth perspective). As libraries release support for alternative protocols and the industry compiles their projects against these newer versions, the adoption of non-FIX will keep increasing. Similar to how you get HTTP/2 for "free" when using the Go stdlib so its downstream consumers such as Caddy and Traefik can provide that as a default to end users. https://en.wikipedia.org/wiki/List_of_electronic_trading_protocols https://en.wikipedia.org/wiki/List_of_electronic_trading_pro... EDIT: Added year for context on timeframe of first implementations

carlivar 5y ago

We love haproxy. It is the load balancer powering millions of requests per second in aggregate in our data centers. These data centers cost less than half what the cloud would.