9 ms·

I'm really on the fence between Haproxy or Nginx. I have used Haproxy successfully in the past, but I'm tempted by the simplicity of Nginx, especially now that

by cmer 14y ago

I'm really on the fence between Haproxy or Nginx. I have used Haproxy successfully in the past, but I'm tempted by the simplicity of Nginx, especially now that it supports SPDY. Would like to hear people's thoughts on using Nginx in "real life" for load balancing rather than Haproxy.

Firehed 14y ago
I can't compare it to HAProxy, but nginx load balancing was probably the simplest and most reliable part of our web infrastructure, and did exactly what we wanted and needed. Never played around with SPDY, but I liked the various options regarding server weighting, SSL termination, and like you mention the ease of configuration. It wasn't too fancy, but solved a problem and solved it well. Unfortunately we had to switch off of it due to PCI compliance concerns[1], but I'd use it again in a heartbeat. [1] not because there were actual issues, but because other solutions were fully audited out of the box. I'm hardly surprised that we've had more issues with those solutions than we ever had with nginx, including the time when we barely knew how to configure the thing. One of the unavoidable hazards of PCI Level 1 :( We still use it for the actual web requests quite happily.
- cmer 14y ago
  Gives me confidence in going with nginx for LB. Was the lack of a web ui (like haproxy has) ever a concern? How did you keep track of dead servers behind the LB?
  - Firehed 14y ago
    We didn't have enough servers behind it to really deal with dead servers, to be honest. Seemed to detect a failed server and route around it quickly enough, and we have monitoring per server in place to go in and reboot the thing or whatever. The configs are pretty straightforward, but might get a little nuts if you're dealing with hundreds of servers behind the thing. I don't have to wear a sysadmin hat too frequently (thank god) but when I did it was pretty easy to deal with. Huge fan of the fact that reloading the config would perform a configtest automatically before trying to apply the new settings. I don't know why all software doesn't do this.
- jaequery 14y ago
  care to elaborate what made it a PCI compliance concern?
  - Firehed 14y ago
    I don't know too many details as I wasn't on that side of the PCI audit (more handling the software we write), but my impression was that off-the-shelf hardware was already certified where nginx was not. It was also one less component for us to manage, as we opted for hosting where we manage our web stack and the hosting company deals with the hardware and network.
- mattdeboard 14y ago
  I love nginx. You stole the words out of my mouth re: simplicity & stability of nginx for load balancing. If there's one thing you can really nail like a pro while still a rookie (like me) it's configuring nginx to load balance.
adrianpike 14y ago
I've been using it heavily under production loads, and the balancing portion hasn't blinked. I'm also doing SSL termination at it, so I don't really have any metrics on the balancing in isolation, but for moving 50-100 concurrent connections around it hasn't blinked. I do really like HAProxy's more flexible up/down monitoring, though. In the past, we've done the trick with separate control connections that we can bring up & down with iptables to shuffle traffic around without any broken connections.
- cmer 14y ago
  Having SSL termination on a single box is definitely a big plus; that's how I wanted to do it as well. Did you miss Haproxy's web ui? Does nginx have any way of reporting if a server is down?
  - sciurus 14y ago
    HAProxy 1.5 supports SSL. I've had good luck with it so far.
    - scottbruin 14y ago
      How much traffic are you pushing through it? I've been planning to set up stunnel + HAProxy on separate instances once we're comfortable going with 1.5, but am curious if we could get away terminating SSL on the same instance as HAProxy runs.
      - sciurus 14y ago
        So far it has just been used in development and QA environments. Unfortunately I'm not sure how much traffic our load testing pushes through it.
meritt 14y ago
You cannot use nginx as a proxy in front of websocket backends currently if you have need for that. nginx 1.3 has it on the roadmap though. HAproxy works correctly for websocket backends today.
- cmer 14y ago
  Hmmm! Good to know. Does 1.3.x unstable have websocket support already? We're using the 1.3 branch in production and it's been super reliable.
- darkarmani 14y ago
  Can either of those solution do dynamic secure web sockets? I want to terminate SSL to various dynamic backend web socket servers. I'm spinning up additional web socket servers per user for user privilege separation.
  - scottbruin 14y ago
    What exactly are you trying to do? Where in the chain are you hoping to terminate SSL? Do you need to inspect the traffic before load balancing it?
    - darkarmani 14y ago
      I'm terminating the SSL outside VMs, so the VMs can be compromised without giving up the certificate's private key. The VMs are each running a websocket server running as the user that will be connecting. This makes the security aspects very easy to handle. Each user can only modify their own environment and write to their own files (backed by unix permissions). Even if they root the VM (excluding hypervisor vulnerabilities) they won't be able to access any private data. If I want to be able to hot migrate VMs between physical machines, I need some way of dynamically proxying the connections. If I had lots of IPs, I could simply let each VM have an IP address and the SSL terminator would route properly no matter where I move the VM. Does that make sense?
      - 14y ago
jvoorhis 14y ago
Nginx has also formed the basis of CloudFoundry's routing tier, and this cloudfoundry.com and appfog.com. Nginx load balancing can be very simple, but you can customize it to your heart's content with Lua [1]. You can also use it for your application tier with Passenger, [U]WSGI, FPM, FCGI... [1] https://github.com/cloudfoundry/cf-release/blob/master/jobs/router/templates/nginx.conf.erb https://github.com/cloudfoundry/cf-release/blob/master/jobs/...
scottbruin 14y ago
What does Nginx offer that HAProxy doesn't? We've been using HAProxy 1.4 for over a year now, up to 1500 req/s on a virtual machine. It's the most reliable piece of all of our infrastructure. Hardest part has been tuning the Linux instance for a lot of connections when encountering DDOS attacks and the like. We also run another HAProxy instance for rate limiting for attacked sites that feeds back into the main load balancer. And this is Layer 7 load balancing including inspecting headers. Never breaks a sweat. 1.5 supports SPDY, which is the last big thing for us (though I need it in the opposite direction from other mentions, used alongside stunnel).
- scottbruin 14y ago
  To add more, as well: we use HAProxy for deploying (tell it to take down nodes using the unix socket), we have it set up to meter requests to a machine just brought back online, and with one command we can route all traffic to a standby Apache instance that serves a maintenance page. On top of that it has the monitoring page, and using the socket we pull stats every minute and ship them off to Librato Metrics as well as watch for high sessions and the like. I (obviously) cannot sing its praises enough.
- cmer 14y ago
  Maybe I'm wrong, but I think Haproxy only supports NPN and not SPDY itself? I'd be delighted if it did support SPDY out of the box!
  - scottbruin 14y ago
    Whoops! I definitely said SPDY and definitely meant the PROXY protocol (http://haproxy.1wt.eu/download/1.5/doc/proxy-protocol.txt http://haproxy.1wt.eu/download/1.5/doc/proxy-protocol.txt) You can setup stunnel to terminate SSL then append this line to the request that's sent to HAProxy, which will then add an X-Forwarded-For header from that info. This may be relevant to your interests, though: http://www.igvita.com/2012/10/31/simple-spdy-and-npn-negotiation-with-haproxy/ http://www.igvita.com/2012/10/31/simple-spdy-and-npn-negotia...

by cmer 14y ago

Firehed 14y ago

I can't compare it to HAProxy, but nginx load balancing was probably the simplest and most reliable part of our web infrastructure, and did exactly what we wanted and needed. Never played around with SPDY, but I liked the various options regarding server weighting, SSL termination, and like you mention the ease of configuration. It wasn't too fancy, but solved a problem and solved it well. Unfortunately we had to switch off of it due to PCI compliance concerns[1], but I'd use it again in a heartbeat. [1] not because there were actual issues, but because other solutions were fully audited out of the box. I'm hardly surprised that we've had more issues with those solutions than we ever had with nginx, including the time when we barely knew how to configure the thing. One of the unavoidable hazards of PCI Level 1 :( We still use it for the actual web requests quite happily.

cmer 14y ago
Gives me confidence in going with nginx for LB. Was the lack of a web ui (like haproxy has) ever a concern? How did you keep track of dead servers behind the LB?
- Firehed 14y ago
  We didn't have enough servers behind it to really deal with dead servers, to be honest. Seemed to detect a failed server and route around it quickly enough, and we have monitoring per server in place to go in and reboot the thing or whatever. The configs are pretty straightforward, but might get a little nuts if you're dealing with hundreds of servers behind the thing. I don't have to wear a sysadmin hat too frequently (thank god) but when I did it was pretty easy to deal with. Huge fan of the fact that reloading the config would perform a configtest automatically before trying to apply the new settings. I don't know why all software doesn't do this.
jaequery 14y ago
care to elaborate what made it a PCI compliance concern?
- Firehed 14y ago
  I don't know too many details as I wasn't on that side of the PCI audit (more handling the software we write), but my impression was that off-the-shelf hardware was already certified where nginx was not. It was also one less component for us to manage, as we opted for hosting where we manage our web stack and the hosting company deals with the hardware and network.
mattdeboard 14y ago
I love nginx. You stole the words out of my mouth re: simplicity & stability of nginx for load balancing. If there's one thing you can really nail like a pro while still a rookie (like me) it's configuring nginx to load balance.

adrianpike 14y ago

I've been using it heavily under production loads, and the balancing portion hasn't blinked. I'm also doing SSL termination at it, so I don't really have any metrics on the balancing in isolation, but for moving 50-100 concurrent connections around it hasn't blinked. I do really like HAProxy's more flexible up/down monitoring, though. In the past, we've done the trick with separate control connections that we can bring up & down with iptables to shuffle traffic around without any broken connections.

cmer 14y ago
Having SSL termination on a single box is definitely a big plus; that's how I wanted to do it as well. Did you miss Haproxy's web ui? Does nginx have any way of reporting if a server is down?
- sciurus 14y ago
  HAProxy 1.5 supports SSL. I've had good luck with it so far.
  - scottbruin 14y ago
    How much traffic are you pushing through it? I've been planning to set up stunnel + HAProxy on separate instances once we're comfortable going with 1.5, but am curious if we could get away terminating SSL on the same instance as HAProxy runs.
    - sciurus 14y ago
      So far it has just been used in development and QA environments. Unfortunately I'm not sure how much traffic our load testing pushes through it.

meritt 14y ago

You cannot use nginx as a proxy in front of websocket backends currently if you have need for that. nginx 1.3 has it on the roadmap though. HAproxy works correctly for websocket backends today.

cmer 14y ago
Hmmm! Good to know. Does 1.3.x unstable have websocket support already? We're using the 1.3 branch in production and it's been super reliable.
darkarmani 14y ago
Can either of those solution do dynamic secure web sockets? I want to terminate SSL to various dynamic backend web socket servers. I'm spinning up additional web socket servers per user for user privilege separation.
- scottbruin 14y ago
  What exactly are you trying to do? Where in the chain are you hoping to terminate SSL? Do you need to inspect the traffic before load balancing it?
  - darkarmani 14y ago
    I'm terminating the SSL outside VMs, so the VMs can be compromised without giving up the certificate's private key. The VMs are each running a websocket server running as the user that will be connecting. This makes the security aspects very easy to handle. Each user can only modify their own environment and write to their own files (backed by unix permissions). Even if they root the VM (excluding hypervisor vulnerabilities) they won't be able to access any private data. If I want to be able to hot migrate VMs between physical machines, I need some way of dynamically proxying the connections. If I had lots of IPs, I could simply let each VM have an IP address and the SSL terminator would route properly no matter where I move the VM. Does that make sense?
    - 14y ago

jvoorhis 14y ago

Nginx has also formed the basis of CloudFoundry's routing tier, and this cloudfoundry.com and appfog.com. Nginx load balancing can be very simple, but you can customize it to your heart's content with Lua [1]. You can also use it for your application tier with Passenger, [U]WSGI, FPM, FCGI... [1] https://github.com/cloudfoundry/cf-release/blob/master/jobs/router/templates/nginx.conf.erb https://github.com/cloudfoundry/cf-release/blob/master/jobs/...

scottbruin 14y ago

What does Nginx offer that HAProxy doesn't? We've been using HAProxy 1.4 for over a year now, up to 1500 req/s on a virtual machine. It's the most reliable piece of all of our infrastructure. Hardest part has been tuning the Linux instance for a lot of connections when encountering DDOS attacks and the like. We also run another HAProxy instance for rate limiting for attacked sites that feeds back into the main load balancer. And this is Layer 7 load balancing including inspecting headers. Never breaks a sweat. 1.5 supports SPDY, which is the last big thing for us (though I need it in the opposite direction from other mentions, used alongside stunnel).

scottbruin 14y ago
To add more, as well: we use HAProxy for deploying (tell it to take down nodes using the unix socket), we have it set up to meter requests to a machine just brought back online, and with one command we can route all traffic to a standby Apache instance that serves a maintenance page. On top of that it has the monitoring page, and using the socket we pull stats every minute and ship them off to Librato Metrics as well as watch for high sessions and the like. I (obviously) cannot sing its praises enough.
cmer 14y ago
Maybe I'm wrong, but I think Haproxy only supports NPN and not SPDY itself? I'd be delighted if it did support SPDY out of the box!
- scottbruin 14y ago
  Whoops! I definitely said SPDY and definitely meant the PROXY protocol (http://haproxy.1wt.eu/download/1.5/doc/proxy-protocol.txt http://haproxy.1wt.eu/download/1.5/doc/proxy-protocol.txt) You can setup stunnel to terminate SSL then append this line to the request that's sent to HAProxy, which will then add an X-Forwarded-For header from that info. This may be relevant to your interests, though: http://www.igvita.com/2012/10/31/simple-spdy-and-npn-negotiation-with-haproxy/ http://www.igvita.com/2012/10/31/simple-spdy-and-npn-negotia...