3 ms·
I'd be interested in hearing what kind of autoscaling system they use for their Ruby pods. We're running a few (legacy — we're moving to Go) Ruby apps in produ
by lobster_johnson 9y ago
I'd be interested in hearing what kind of autoscaling system they use for their Ruby pods.
We're running a few (legacy — we're moving to Go) Ruby apps in production on Kubernetes. We're using Puma, which is very similar to Unicorn, and it's unclear what the optimal strategy here is. I've not benchmarked this in any systematic way.
For example, in theory you could make a single deployment run a single Unicorn worker, then set resources:requests:cpu and resources:limits:cpu both to 1.0, and then add a horizontal pod autoscaler that's set to scale the deployment up on, say, 80% CPU.
But that gives you terrible request rates, and will be choking long before it's reaching 80% CPU. So it's better to give it, say, 4 workers. At the same time, it's counter-productive to allocate it 4 CPUs, because Ruby will generally not be able to utilize them fully. At the same time, more workers mean a lot more memory usage, obviously.
I did some quick benchmarking, and found I could give them 4 workers but still constrain to 1 CPU, and that would still give me a decent qps.
- sytse 9y agoAt GitLab we recommend to use CPU cores + 1 as the number of unicorn workers https://docs.gitlab.com/ce/install/requirements.html#unicorn-workers https://docs.gitlab.com/ce/install/requirements.html#unicorn...
- lobster_johnson 9y agoHow do you configure that? A pod doesn't know what machine it's running on ahead of time. You can create nodepools and use node selectors to pin the pod to that nodepool, but I'm not sure I love the idea.
- ecnahc515 9y agoUsing the downward API, your application can get the number of CPU cores or the millicore value via environment variables or volumes. This would let you configure your number of workers based on what resource limits you set on your container.
- twk3 9y agoOur entrypoint configures the unicorn workers before starting using a chef omnibus call. So we grab the memory and cpus using Ohai: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/attributes/default.rb#L303 https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/fil... and do cpus+1 ^This is pretty much we do by default for all our regular package installs, but in some of our Kubernetes Helm charts we instead statically configure the pod resources and unicorn workers. (Defaulting to 1cpu - 2workers per front-end pod) eg: https://gitlab.com/charts/charts.gitlab.io/blob/master/charts/gitlab/values.yaml#L198 https://gitlab.com/charts/charts.gitlab.io/blob/master/chart... As someone mentioned in this thread, using the downward api might be a cool way to configure the workers.