4 ms·

Here's a story that might make you not want to do that. We ran Kubernetes with the standard scheduler and node autoscaling for a long time, and used to allow d

Here's a story that might make you not want to do that. We ran Kubernetes with the standard scheduler and node autoscaling for a long time, and used to allow developers in our (simplified) manifests define resource requests and limits. We saw that with our current config, we always had some unused capacity (that we wanted) since the scheduler spread out workloads while the autoscaler was only throwing nodes away with less than 70% load. So we started ignoring the limits provided by developers. This was initially a great success, our response times in the 99th went down drastically, even during sudden traffic spikes. 2 years later, and nobody cares about resource allocation for new services anymore. We can essentially never disable bursting again, because too many services (100+) use the extra capacity constantly, and due to our organizational structure we can't really _make_ these teams fix their allocations.

nrmitchi 6y ago
As an unsolicited recommendation, if you _wanted_ to attempt to chase this down, you should look at using the Vertical Pod Autoscaler (https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler https://github.com/kubernetes/autoscaler/tree/master/vertica...) in recommendation only mode in order to get a good idea of what services are actually using, and set new limits based on that. It definitely won't be optimal, but should let you get to a place where you at least have some limits set.
dilyevsky 6y ago
We at Mux have removed nearly all limits but setup alerts that trigger when container consistently bursts above the request so we can chase those down (temporary bursts are ignored). Never had any issues
- sascha_sl 6y ago
  This might work if your developer (who do not have to care about operational things at all) to SRE (and not even dedicated SRE) ratio is not 30... Point being, if you don't have the capability to somehow keep teams in check through process and not pure capability, reconsider.
  - dilyevsky 6y ago
    We enforce cpu request (and memory request and limit) via process and plan on adding automation to enforce that so shouldn’t be a problem even with scaled up team since you can only hurt yourself by setting request too low. Not sure how the number 30 was chosen...

nrmitchi 6y ago

As an unsolicited recommendation, if you _wanted_ to attempt to chase this down, you should look at using the Vertical Pod Autoscaler (https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler https://github.com/kubernetes/autoscaler/tree/master/vertica...) in recommendation only mode in order to get a good idea of what services are actually using, and set new limits based on that. It definitely won't be optimal, but should let you get to a place where you at least have some limits set.

dilyevsky 6y ago

We at Mux have removed nearly all limits but setup alerts that trigger when container consistently bursts above the request so we can chase those down (temporary bursts are ignored). Never had any issues

sascha_sl 6y ago
This might work if your developer (who do not have to care about operational things at all) to SRE (and not even dedicated SRE) ratio is not 30... Point being, if you don't have the capability to somehow keep teams in check through process and not pure capability, reconsider.
- dilyevsky 6y ago
  We enforce cpu request (and memory request and limit) via process and plan on adding automation to enforce that so shouldn’t be a problem even with scaled up team since you can only hurt yourself by setting request too low. Not sure how the number 30 was chosen...