4 ms·

Scaling Down Gracefully with AWS Lambda and HAProxy

65 points by realityking 10y ago

bracki 10y ago
So what do you need haproxy for that an ELB cannot handle?
andonaeus 10y ago
Interesting article. While my company's cloud infrastructure is in its final nascent stages, I will definitely share this article and the ideas therein for basically reloading the HAProxy context w.r.t the proxied backends.
atom_enger 10y ago
How do you ensure that no requests are lost when you reload HAProxy?
- jvehent 10y ago
  Soft reload takes care of that for you, sockets aren't dropped. If you restart, however, everything's dropped.
  - atom_enger 10y ago
    Is that new in 1.6? As of 1.5.11 there was a lot of hacking required to get a true zero downtime reload: http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html http://engineeringblog.yelp.com/2015/04/true-zero-downtime-h...
    - luhn 10y ago
      No, he's referring to the soft reload added in 1.5, which the article you linked addresses under "Just Reload HAProxy." As the article points out, it's not quite zero-downtime, but pretty close.
      - jontro 10y ago
        Why not drop servers in a way described here? http://serverfault.com/questions/249316/how-can-i-remove-balanced-node-from-haproxy-via-command-line http://serverfault.com/questions/249316/how-can-i-remove-bal...
        jolynch 10y ago
        You can down servers or backends over the stats socket, but you can't add them (adding servers requires a reload). The DNS resolution work that is going into 1.6 releases may lead to a workable approach where you over allocate servers in a backend and then update them to have the right host/port. Last I checked, however, you still couldn't set the port dynamically.
totally 10y ago
So HA only updates when chef runs (every 15 mins or so)? And when nodes are deleted, I guess something else has to clean them from chef? You'll also have to add these hosts to monitoring config. I implemented something similar with puppet/mcollective/SQS years back. Poor man's service discovery. Here are some other options that, if you can handle the complexity of running Zookeper (or similar), are probably a bit cleaner: http://jasonwilder.com/blog/2014/02/04/service-discovery-in-the-cloud/ http://jasonwilder.com/blog/2014/02/04/service-discovery-in-...
- deleted 10y ago
  [deleted]
angryasian 10y ago
What are the conditions that using HAProxy is preferable over using ELB or in conjunction with ?
- toomuchtodo 10y ago
  When you want more control over your load balancing, or you have long lived connections that exceed the connection timeout you can configure with an ELB (RabbitMQ connections, for example; ask if you want deets, we've tried using ELBs multiple times with Rabbit, and always come back to HAproxy).
  - bmurphy1976 10y ago
    Haproxy also supports mutual SSL and can do content-based health checks. Logs don't have to go through an S3 bucket so you can also get real-time metrics and logging if you want. Also, because you have control over it, you have more options for service discovery (i.e. write a Python script that periodically queries a Kubernetes cluster). And virtual hosts... And SNI... And about a thousand other features. Honestly, ELBs are pretty crappy.
    - toomuchtodo 10y ago
      ELBs are awesome if you need stupid, simple load balancing, and aren't too picky about how they behave. Anything else? Haproxy. I love that I can specify existing IAM SSL certs to an ELB; I don't have to provide access to the actual certs/keys to folks who want to provision their own ELBs in our environment, they simply need to pass through the IAM resource in the config. Its a small thing, but still important. EDIT: Something to note though, from my trails of broken glass I've walked; the process limits haproxy displays in the web interface don't always match the process limits the process actually has (as determined via /proc/<process_id>/limits). Don't get burned like I did (too few file descriptors despite haproxy indicating the contrary, epic connection failures).
  - whalesalad 10y ago
    We had issues with internal ELB's for RabbitMQ (the 1 hour idle connection timeout) and the solution ended up being a heartbeat in the clients. Most client libraries have a parameter that will perform a heartbeat at some interval to keep the connection open. I can't imagine deploying our own HAProxy infrastructure. Sure, it's relatively simple, but the set-it-and-forget-it comfort provided by ELB (not to mention free SSL) is hard to beat.

65 points by realityking 10y ago

So what do you need haproxy for that an ELB cannot handle?

Interesting article. While my company's cloud infrastructure is in its final nascent stages, I will definitely share this article and the ideas therein for basically reloading the HAProxy context w.r.t the proxied backends.

atom_enger 10y ago

How do you ensure that no requests are lost when you reload HAProxy?

jvehent 10y ago
Soft reload takes care of that for you, sockets aren't dropped. If you restart, however, everything's dropped.
- atom_enger 10y ago
  Is that new in 1.6? As of 1.5.11 there was a lot of hacking required to get a true zero downtime reload: http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html http://engineeringblog.yelp.com/2015/04/true-zero-downtime-h...
  - luhn 10y ago
    No, he's referring to the soft reload added in 1.5, which the article you linked addresses under "Just Reload HAProxy." As the article points out, it's not quite zero-downtime, but pretty close.
    - jontro 10y ago
      Why not drop servers in a way described here? http://serverfault.com/questions/249316/how-can-i-remove-balanced-node-from-haproxy-via-command-line http://serverfault.com/questions/249316/how-can-i-remove-bal...
      - jolynch 10y ago
        You can down servers or backends over the stats socket, but you can't add them (adding servers requires a reload). The DNS resolution work that is going into 1.6 releases may lead to a workable approach where you over allocate servers in a backend and then update them to have the right host/port. Last I checked, however, you still couldn't set the port dynamically.

totally 10y ago

So HA only updates when chef runs (every 15 mins or so)? And when nodes are deleted, I guess something else has to clean them from chef? You'll also have to add these hosts to monitoring config. I implemented something similar with puppet/mcollective/SQS years back. Poor man's service discovery. Here are some other options that, if you can handle the complexity of running Zookeper (or similar), are probably a bit cleaner: http://jasonwilder.com/blog/2014/02/04/service-discovery-in-the-cloud/ http://jasonwilder.com/blog/2014/02/04/service-discovery-in-...

deleted 10y ago
[deleted]

angryasian 10y ago

What are the conditions that using HAProxy is preferable over using ELB or in conjunction with ?

toomuchtodo 10y ago
When you want more control over your load balancing, or you have long lived connections that exceed the connection timeout you can configure with an ELB (RabbitMQ connections, for example; ask if you want deets, we've tried using ELBs multiple times with Rabbit, and always come back to HAproxy).
- bmurphy1976 10y ago
  Haproxy also supports mutual SSL and can do content-based health checks. Logs don't have to go through an S3 bucket so you can also get real-time metrics and logging if you want. Also, because you have control over it, you have more options for service discovery (i.e. write a Python script that periodically queries a Kubernetes cluster). And virtual hosts... And SNI... And about a thousand other features. Honestly, ELBs are pretty crappy.
  - toomuchtodo 10y ago
    ELBs are awesome if you need stupid, simple load balancing, and aren't too picky about how they behave. Anything else? Haproxy. I love that I can specify existing IAM SSL certs to an ELB; I don't have to provide access to the actual certs/keys to folks who want to provision their own ELBs in our environment, they simply need to pass through the IAM resource in the config. Its a small thing, but still important. EDIT: Something to note though, from my trails of broken glass I've walked; the process limits haproxy displays in the web interface don't always match the process limits the process actually has (as determined via /proc/<process_id>/limits). Don't get burned like I did (too few file descriptors despite haproxy indicating the contrary, epic connection failures).
- whalesalad 10y ago
  We had issues with internal ELB's for RabbitMQ (the 1 hour idle connection timeout) and the solution ended up being a heartbeat in the clients. Most client libraries have a parameter that will perform a heartbeat at some interval to keep the connection open. I can't imagine deploying our own HAProxy infrastructure. Sure, it's relatively simple, but the set-it-and-forget-it comfort provided by ELB (not to mention free SSL) is hard to beat.