11 ms·

HAProxy: How to temporary disable a back end server using the command line

104 points by Napsty 4y ago

sposeray 4y ago
[dead]
Amfy 4y ago
Error 403. Did he disable the last back end server in the pool? See info below, correct link: https://www.claudiokuenzler.com/blog/1240/haproxy-how-to-disable-enable-backend-server-command-line-cli-socket https://www.claudiokuenzler.com/blog/1240/haproxy-how-to-dis...
- Napsty 4y ago
  I did (I am the owner of the blog). The blog article is on www.claudiokuenzler.com - not on this kakoku URL. I have no idea why Hacker News would change the link to another domain. Maybe someone knows this?
  - Amfy 4y ago
    email dang (hn@ycombinator.com). He and his team will be able to fix this in no time.
    - Napsty 4y ago
      Thanks for the hint! It may have been my error as well as I just discovered that copy cat URL before and blocked it. May have posted the copy-cat URL myself, who knows. Anyway I just posted a new submission with the correct link.
      - dang 4y ago
        Changed now from https://gby.kakoku.online/blog/1240/haproxy-how-to-disable-enable-backend-server-command-line-cli-socket https://gby.kakoku.online/blog/1240/haproxy-how-to-disable-e.... Thanks to both of you! Our software does change links sometimes (it follows redirects and also uses the canonical URL when it finds one), but from the logs it doesn't look like any of that happened here.
        Napsty 4y ago
        thx, appreciate it! still need to figure out what this okaku domain does though...
deleted 4y ago
[deleted]
bigoldie 4y ago
Or use HATop :-)
- mtmail 4y ago
  Last updated 2010. I'm all for software that doesn't need new features but no changed in a decade make me not want to install it on a production server http://feurix.org/projects/hatop/ http://feurix.org/projects/hatop/
crizzlenizzle 4y ago
Correct link: https://www.claudiokuenzler.com/blog/1240/haproxy-how-to-disable-enable-backend-server-command-line-cli-socket https://www.claudiokuenzler.com/blog/1240/haproxy-how-to-dis...
- Napsty 4y ago
  thx mate! just added the correct link as a new submission, too.
- dang 4y ago
  Fixed now. Thanks!
Hackerpronoov 4y ago
simonjgreen 4y ago
hatop is a great ncurses interface to this from cli. https://github.com/feurix/hatop https://github.com/feurix/hatop Can also be done remotely https://github.com/Wirehive/haproxy-remote https://github.com/Wirehive/haproxy-remote
- tomputer 4y ago
  Indeed! HATop is great to show the status, traffic, HTTP codes, errors or number of connections. It is also very simple to enable/disable HAProxy backends, for example: hatop -s /var/run/haproxy/example.sock Use the arrows to select a backend server and press: F9 - Enable a backend server (Status: UP) F10 - Disable a backend server (Status: MAINTENANCE) For Apple keyboards it is fn+F9 and fn+F10.
deleted 4y ago
[deleted]
mihaigalos 4y ago
I prefer commenting out the entry referencing the server/node directly in the haproxy.cfg or even better, the code used to generate it und version control (Config-as-Code).
- Napsty 4y ago
  Yes but a reload can still cut a KEEP ALIVE established connection (observed in older HAProxy 1.6). Probably solved in more recent versions though.
  - bsagdiyev 4y ago
    This may be resolved in newer versions but definitely bit me before in older ones. I just have a script that drains off the backends we need before putting them in maintenance mode. Makes working on individual hosts much easier.
  - endre 4y ago
    also, old haproxy process (with opened connections) still does checks which might be undesirable at scale. as such we try to avoid reloading haproxy as much as possible. you can even renew a cert on-the-fly uploading the new cert via the admin socket.
- linsomniac 4y ago
  Seems a little heavy weight for my "remove from LB, update code, add back to LB" Ansible script. :-)
  - hayst4ck 4y ago
    An alternative I like a bit more than, `remove, update, add back` is to set a signal handler to close the listening socket, wait for current requests to finish, and then exit. Assuming you initialize every service you depend on at program start and not in the critical path and that health checks only function after this setup work has been done it simplifies operations a bit and decreases the liklhihood of configuration errors (a drained server in a load balancer) from persisting.
darkwater 4y ago
The irony: ``` 403 Forbidden nginx ```
- taf2 4y ago
  IMO not really - a valid architecture is nginx - haproxy - app servers
  - darkwater 4y ago
    IME haproxy and nginx can overlap a lot as balancers/rewriter; having both of them adds no real world benefits, unless you have some special case where you need one special, unique feature from both of them.
    - hayst4ck 4y ago
      > having both of them adds no real world benefits This depends a great deal on the architecture of your application server and traffic load. There are a lot of reasons that it is nicer to use a queue in front of your load balancer, rather than use queues in your application server, or worse, use the operating system queue. You must have SSL termination, logging, load balancing, and queuing. You probably want to modify headers and interpret cookies as well. HAProxy is a queue that feeds into a load balancer. Nginx is a buffer that feeds into a load balancer (IIRC). One useful property that HAProxy can ensure is no more than 1 request going to a server at a time. I believe (but potentially incorrectly) that nginx will fail requests, rather than "queue" them, if it tries to ensure this property. HAProxy also historically had health checks for backend servers, while for nginx it was a paid premium feature. I could be wrong about nginx not being able to queue. It has been a while. For my mental model, nginx is the right choice for SSL termination, logging, request mangling, interpretation of cookies and loadbalancing based on request information (for example choosing haproxy instances based on a domain name). HAProxy is the right tool for queuing and load balancing to servers that will actually fulfill the request.
      - TimWolla 4y ago
philliphaydon 4y ago
There’s other ways than using telnet??? I thought telnet was the recommended way.
vinay_ys 4y ago
Haproxy is definitely under-appreciated for the amount of heavy lifting it does in a high-scale deployment.
- NDizzle 4y ago
  A lot of just don't realize how and where it's used. I'm almost 100% positive that Microsoft uses HAProxy (open source or commercial license I couldn't tell you) to power App Services on Azure. Yet you'll get friction from CTOs at "Microsoft shops" for choosing HAProxy to use yourself.
  - teh_klev 4y ago
    > I'm almost 100% positive that Microsoft uses HAProxy (open source or commercial license I couldn't tell you) to power App Services on Azure. I think you'll find they're using ARR and not HAProxy for the "batteries included" load balancer. You can tell this from the cookie names "ARRAffinity" and "ARRAffinitySameSite".
  - Napsty 4y ago
    Almost certain that AWS uses HAProxy behind their ELB (Elastic Load Balancer) service. But if you know HAProxy, you feel very limited when using ELB.
lukeqsee 4y ago
We built a fantastic CDN on top of HAProxy, and all these nifty tricks are only scratching the surface of its potential. The ease with which you can make real-time reactions to threats and conditions is really second to none, and with the ability to truly "hitless" reload (reload without affecting in-flight requests and connections) it makes those have immediate impact and no customer impact. Sometimes the capabilities make the config a bit inscrutable, but other than that, I can't recommend it enough.
- xani_ 4y ago
  Newer versions also have few more tricks up the sleeve, like you can probe and act upon client's TCP RTT and RTT variance (because apparently newer kernels just track that info and it can be accessed from userspace). We ended up using HAPRoxy as front for pretty much everything (even if techncially slapping just Nginx would be simpler at first), just because it often happens that this and that needs to be added (stuff like adding proper headers when devs don't want to figure out how to make their '00s CMS emi the '20s security headers). It can also talk directly via unix socket to the backends so you don't even need to have a bunch of ports open if you want to front something.
  - pull_my_finger 4y ago
    Have you used the ngx_lua module (or OpenResty) with Nginx at all? I'm curious if there was something HAProxy does that can't be done with OpenResty, because that is crazy full-featured. Basically the main reason I haven't played around with HAProxy is just because I haven't ever felt like I was missing anything with OpenResty/Nginx
    - lukeqsee 4y ago
      HAProxy also has Lua support. The general answer here, is no, I don't think you can do much in HAProxy that you can't do with OpenResty, but with HAProxy you generally don't have to pay the overhead of Lua (and HAProxy tends to be very well optimized, so the difference can be significant at high throughput of requests / bytes).
jabart 4y ago
HAProxy also has the data plane api, which is another process that exposes an API that lets you do the same thing but remote and over a rest api.
- manishsharan 4y ago
  Dataplane API is pretty sweet : https://www.haproxy.com/documentation/dataplaneapi/community/#get-/services/haproxy/configuration/acls https://www.haproxy.com/documentation/dataplaneapi/community...
  - ElevenLathe 4y ago
    Doesn't "Dataplane API" actually constitute part of the control plane?
bobek 4y ago
HAproxy is awesome. We have used the socket for ingress rate limiter for reasonably successful VoD service.
- elforce002 4y ago
  Wow. Nice. I'm working on a similar project atm. Do you have any info on how to configure the socket?
xani_ 4y ago
you can also add stats admin to the stats backend to enable ability to do that from statistics UI address. Just make sure it is at the very least behind a password
rglover 4y ago
HAProxy has blown my mind. I'm using it to build a deployment tool for Joystick [1] and it's a real treat to work with. I utilize this specific feature (enabling/disabling servers on the fly) and it worked on the first swing without issue. Made scaling a cluster of servers fairly effortless. [1] https://github.com/cheatcode/joystick https://github.com/cheatcode/joystick

104 points by Napsty 4y ago

[dead]

Amfy 4y ago

Error 403. Did he disable the last back end server in the pool? See info below, correct link: https://www.claudiokuenzler.com/blog/1240/haproxy-how-to-disable-enable-backend-server-command-line-cli-socket https://www.claudiokuenzler.com/blog/1240/haproxy-how-to-dis...

Napsty 4y ago
I did (I am the owner of the blog). The blog article is on www.claudiokuenzler.com - not on this kakoku URL. I have no idea why Hacker News would change the link to another domain. Maybe someone knows this?
- Amfy 4y ago
  email dang (hn@ycombinator.com). He and his team will be able to fix this in no time.
  - Napsty 4y ago
    Thanks for the hint! It may have been my error as well as I just discovered that copy cat URL before and blocked it. May have posted the copy-cat URL myself, who knows. Anyway I just posted a new submission with the correct link.
    - dang 4y ago
      Changed now from https://gby.kakoku.online/blog/1240/haproxy-how-to-disable-enable-backend-server-command-line-cli-socket https://gby.kakoku.online/blog/1240/haproxy-how-to-disable-e.... Thanks to both of you! Our software does change links sometimes (it follows redirects and also uses the canonical URL when it finds one), but from the logs it doesn't look like any of that happened here.
      - Napsty 4y ago
        thx, appreciate it! still need to figure out what this okaku domain does though...

deleted 4y ago

[deleted]

bigoldie 4y ago

Or use HATop :-)

mtmail 4y ago
Last updated 2010. I'm all for software that doesn't need new features but no changed in a decade make me not want to install it on a production server http://feurix.org/projects/hatop/ http://feurix.org/projects/hatop/

crizzlenizzle 4y ago

Correct link: https://www.claudiokuenzler.com/blog/1240/haproxy-how-to-disable-enable-backend-server-command-line-cli-socket https://www.claudiokuenzler.com/blog/1240/haproxy-how-to-dis...

Napsty 4y ago
thx mate! just added the correct link as a new submission, too.
dang 4y ago
Fixed now. Thanks!

Hackerpronoov 4y ago

simonjgreen 4y ago

hatop is a great ncurses interface to this from cli. https://github.com/feurix/hatop https://github.com/feurix/hatop Can also be done remotely https://github.com/Wirehive/haproxy-remote https://github.com/Wirehive/haproxy-remote

tomputer 4y ago
Indeed! HATop is great to show the status, traffic, HTTP codes, errors or number of connections. It is also very simple to enable/disable HAProxy backends, for example: hatop -s /var/run/haproxy/example.sock Use the arrows to select a backend server and press: F9 - Enable a backend server (Status: UP) F10 - Disable a backend server (Status: MAINTENANCE) For Apple keyboards it is fn+F9 and fn+F10.

deleted 4y ago

[deleted]

mihaigalos 4y ago

I prefer commenting out the entry referencing the server/node directly in the haproxy.cfg or even better, the code used to generate it und version control (Config-as-Code).

Napsty 4y ago
Yes but a reload can still cut a KEEP ALIVE established connection (observed in older HAProxy 1.6). Probably solved in more recent versions though.
- bsagdiyev 4y ago
  This may be resolved in newer versions but definitely bit me before in older ones. I just have a script that drains off the backends we need before putting them in maintenance mode. Makes working on individual hosts much easier.
- endre 4y ago
  also, old haproxy process (with opened connections) still does checks which might be undesirable at scale. as such we try to avoid reloading haproxy as much as possible. you can even renew a cert on-the-fly uploading the new cert via the admin socket.
linsomniac 4y ago
Seems a little heavy weight for my "remove from LB, update code, add back to LB" Ansible script. :-)
- hayst4ck 4y ago
  An alternative I like a bit more than, `remove, update, add back` is to set a signal handler to close the listening socket, wait for current requests to finish, and then exit. Assuming you initialize every service you depend on at program start and not in the critical path and that health checks only function after this setup work has been done it simplifies operations a bit and decreases the liklhihood of configuration errors (a drained server in a load balancer) from persisting.

darkwater 4y ago

The irony: ``` 403 Forbidden nginx ```

taf2 4y ago
IMO not really - a valid architecture is nginx - haproxy - app servers
- darkwater 4y ago
  IME haproxy and nginx can overlap a lot as balancers/rewriter; having both of them adds no real world benefits, unless you have some special case where you need one special, unique feature from both of them.
  - hayst4ck 4y ago
    > having both of them adds no real world benefits This depends a great deal on the architecture of your application server and traffic load. There are a lot of reasons that it is nicer to use a queue in front of your load balancer, rather than use queues in your application server, or worse, use the operating system queue. You must have SSL termination, logging, load balancing, and queuing. You probably want to modify headers and interpret cookies as well. HAProxy is a queue that feeds into a load balancer. Nginx is a buffer that feeds into a load balancer (IIRC). One useful property that HAProxy can ensure is no more than 1 request going to a server at a time. I believe (but potentially incorrectly) that nginx will fail requests, rather than "queue" them, if it tries to ensure this property. HAProxy also historically had health checks for backend servers, while for nginx it was a paid premium feature. I could be wrong about nginx not being able to queue. It has been a while. For my mental model, nginx is the right choice for SSL termination, logging, request mangling, interpretation of cookies and loadbalancing based on request information (for example choosing haproxy instances based on a domain name). HAProxy is the right tool for queuing and load balancing to servers that will actually fulfill the request.
    - TimWolla 4y ago

philliphaydon 4y ago

There’s other ways than using telnet??? I thought telnet was the recommended way.

vinay_ys 4y ago

Haproxy is definitely under-appreciated for the amount of heavy lifting it does in a high-scale deployment.

NDizzle 4y ago
A lot of just don't realize how and where it's used. I'm almost 100% positive that Microsoft uses HAProxy (open source or commercial license I couldn't tell you) to power App Services on Azure. Yet you'll get friction from CTOs at "Microsoft shops" for choosing HAProxy to use yourself.
- teh_klev 4y ago
  > I'm almost 100% positive that Microsoft uses HAProxy (open source or commercial license I couldn't tell you) to power App Services on Azure. I think you'll find they're using ARR and not HAProxy for the "batteries included" load balancer. You can tell this from the cookie names "ARRAffinity" and "ARRAffinitySameSite".
- Napsty 4y ago
  Almost certain that AWS uses HAProxy behind their ELB (Elastic Load Balancer) service. But if you know HAProxy, you feel very limited when using ELB.

lukeqsee 4y ago

We built a fantastic CDN on top of HAProxy, and all these nifty tricks are only scratching the surface of its potential. The ease with which you can make real-time reactions to threats and conditions is really second to none, and with the ability to truly "hitless" reload (reload without affecting in-flight requests and connections) it makes those have immediate impact and no customer impact. Sometimes the capabilities make the config a bit inscrutable, but other than that, I can't recommend it enough.

xani_ 4y ago
Newer versions also have few more tricks up the sleeve, like you can probe and act upon client's TCP RTT and RTT variance (because apparently newer kernels just track that info and it can be accessed from userspace). We ended up using HAPRoxy as front for pretty much everything (even if techncially slapping just Nginx would be simpler at first), just because it often happens that this and that needs to be added (stuff like adding proper headers when devs don't want to figure out how to make their '00s CMS emi the '20s security headers). It can also talk directly via unix socket to the backends so you don't even need to have a bunch of ports open if you want to front something.
- pull_my_finger 4y ago
  Have you used the ngx_lua module (or OpenResty) with Nginx at all? I'm curious if there was something HAProxy does that can't be done with OpenResty, because that is crazy full-featured. Basically the main reason I haven't played around with HAProxy is just because I haven't ever felt like I was missing anything with OpenResty/Nginx
  - lukeqsee 4y ago
    HAProxy also has Lua support. The general answer here, is no, I don't think you can do much in HAProxy that you can't do with OpenResty, but with HAProxy you generally don't have to pay the overhead of Lua (and HAProxy tends to be very well optimized, so the difference can be significant at high throughput of requests / bytes).

jabart 4y ago

HAProxy also has the data plane api, which is another process that exposes an API that lets you do the same thing but remote and over a rest api.

manishsharan 4y ago
Dataplane API is pretty sweet : https://www.haproxy.com/documentation/dataplaneapi/community/#get-/services/haproxy/configuration/acls https://www.haproxy.com/documentation/dataplaneapi/community...
- ElevenLathe 4y ago
  Doesn't "Dataplane API" actually constitute part of the control plane?

bobek 4y ago

HAproxy is awesome. We have used the socket for ingress rate limiter for reasonably successful VoD service.

elforce002 4y ago
Wow. Nice. I'm working on a similar project atm. Do you have any info on how to configure the socket?

xani_ 4y ago

you can also add stats admin to the stats backend to enable ability to do that from statistics UI address. Just make sure it is at the very least behind a password

rglover 4y ago

HAProxy has blown my mind. I'm using it to build a deployment tool for Joystick [1] and it's a real treat to work with. I utilize this specific feature (enabling/disabling servers on the fly) and it worked on the first swing without issue. Made scaling a cluster of servers fairly effortless. [1] https://github.com/cheatcode/joystick https://github.com/cheatcode/joystick