4 ms·

Our use case is not for scaling the cluster up and down, but rather for ensuring cluster size and stability. When AWS nodes randomly go away or are retired, etc

by crad 10y ago

Our use case is not for scaling the cluster up and down, but rather for ensuring cluster size and stability. When AWS nodes randomly go away or are retired, etc, the autoscaling group will ensure that we keep N nodes in the cluster and they're up and running by using ELB health checking. This is the first version that has the destructive action of removing nodes from a cluster when they disappear and even then it's behind two different config variables: One to turn cluster health checking on and the other to actually do something when bad nodes are noticed. Bad nodes being nodes that are no longer visible in the service discovery backend and no longer pingable by the cluster.

spotman 10y ago
So if you have an ASG of 4 nodes and one goes down, how do you know that the traffic will land on the right node? What if that node is really busy? When the new machine comes up automatically does the system migrate queue ownership to it? Is there a time when the system can be partially unavailable? Possibly most importantly would be, how are you routing queues to hosts and how is the platform load balanced. Using ELB and a tcp listener is a great way to throw away performance with clustered rabbit, not to mention adding a ceiling as to how wide the cluster can get.
- crad 10y ago
  In our case, we use queues in ha-mode:all and queue ownership isn't as much of an issue there. Our biggest limiter for some of our workloads is actually message size, not velocity. That may change in the future. You're absolutely correct re load balancing for high-performance/ceilings/etc. It's not a huge issue for our use, but I can how it can be for others. In my previous environment, for example, it was absolutely critical. I'm sure this tool isn't for everyone, for us it was about making cluster startup easier and letting RabbitMQ handle that instead of a configuration management system like Chef/etc. For example, our test environment is torn down every night on an autoscaling schedule and rebuilt in the morning. Not having to worry about config management for starting up a new RabbitMQ cluster on demand is a nice feature. :) Thanks for pointing out the clear (and accurate) concerns one must consider when running RabbitMQ @ scale. It sounds your experiences are similar to my own with where the gotchas can be.

by crad 10y ago

spotman 10y ago

So if you have an ASG of 4 nodes and one goes down, how do you know that the traffic will land on the right node? What if that node is really busy? When the new machine comes up automatically does the system migrate queue ownership to it? Is there a time when the system can be partially unavailable? Possibly most importantly would be, how are you routing queues to hosts and how is the platform load balanced. Using ELB and a tcp listener is a great way to throw away performance with clustered rabbit, not to mention adding a ceiling as to how wide the cluster can get.

crad 10y ago
In our case, we use queues in ha-mode:all and queue ownership isn't as much of an issue there. Our biggest limiter for some of our workloads is actually message size, not velocity. That may change in the future. You're absolutely correct re load balancing for high-performance/ceilings/etc. It's not a huge issue for our use, but I can how it can be for others. In my previous environment, for example, it was absolutely critical. I'm sure this tool isn't for everyone, for us it was about making cluster startup easier and letting RabbitMQ handle that instead of a configuration management system like Chef/etc. For example, our test environment is torn down every night on an autoscaling schedule and rebuilt in the morning. Not having to worry about config management for starting up a new RabbitMQ cluster on demand is a nice feature. :) Thanks for pointing out the clear (and accurate) concerns one must consider when running RabbitMQ @ scale. It sounds your experiences are similar to my own with where the gotchas can be.