4 ms·

How are the limits incorporated into scheduling? I assumed that was based on requests. What does "scale atomically" even mean? How does removing limits relate

How are the limits incorporated into scheduling? I assumed that was based on requests. What does "scale atomically" even mean? How does removing limits relate to horizontal vs vertical? HPA is based on request utilization, not limits, afaik. What's your take on the arguments against limits in the comment at https://news.ycombinator.com/item?id=24356073 https://news.ycombinator.com/item?id=24356073 ?

tpxl 6y ago
>How does removing limits relate to horizontal vs vertical? Vertical -> give more resources to the program Horizontal -> run more instances of the program Removing limits gives your pods more resources (scaling them vertically) whereas creating more pods creates more copies (scaling horizontally). Assuming parent meant scaling by whole units with "scale atomically", that is you have one or two running programs, not "1.5" if you just give it 50% more resources.
- oso2k 6y ago
  tpxl gets me. :D Even the "scale atomically" part. People seem to have inferred that I believe that Limits are used by the Scheduler. I don't. But if we set "Request = Limits", we're guaranteeing to the Scheduler that our pod workload will never need more than what is Requested, or, we scale up to a new pod. It seems to me latency is a symptom of the actual issue, not the actual problem. If a workload idles at 25% of Request, 12.5% of Limits (as in TFA), and peaks at 50% of Request, 25% of Limits that seems hugely wasteful. What's more, the workload has several "opportunities" to optimize latency. And uncapping the CPU Limit reduces the latency. If it were me, I'd be asking, "Why does my workload potentially need access to (but not utilization?) 4, 6, 8, 16, 32 cores to reduce its latency?" More often than not, I've been able to help customers reduce their latency by DECREASING the Pod's Requests and Limits, but also INCREASE the replica count (via HPA or manually). It's not a silver bullet, and whether a workload is node.js, JBoss EAP, Spring Boot, or Quarkus does matter to some extent. The first thing I reach for in my k8s toolbox is to scale out. "Many hands make light work" is an old adage. N+1 workloads can usually respond to more traffic than N workloads in a shorter amount of time. k8s' strength is that it is networked and clustered. Forcing one node or a set of nodes to work harder (TFA mentions "isolating" the workload) or vertically scaling is anti-pattern in my book. Especially when you understand the workload pattern well. What is being done here is that nodes (which are likely VMs) are being over-committed [0]. Now, those VMs live on physical hypervisors which are likely -guess what- over-committed. Turtles of (S)POFs all the way down I say. Also, TFA mentions In the past we’ve seen some nodes going to a "notReady" state, mainly because some services were using too much resources in a node. and The downsides are that we lose in “container density”, the number of containers that can run in a single node. We could also end up with a lot of “slack” during a low traffic time. You could also hit some high CPU usage, but nodes autoscaling should help you with it. So they acknowledge the risk is real and they've encountered it. For most of my customers, failing nodes, reduced "container density", and "slack" are unacceptable. That translates into increased engineer troubleshooting time, higher cloud provider bills. What's worse is that the suggestion of the Cluster Autoscaler will protect you also comes with increased costs (licenses, VM, storage, etc.). Not the solution I want. Seems like a blank check to your cloud provider. But I get it. I've fought with customers that tell me, "By removing the Limit, my container starts up in half the time." Great. Then they get to Perf Testing and they get wildly inconsistent speed up when scaling out (or way sublinear), or they're limited by resource in their ability to scale up especially when metrics tells them they have resources available, or there is unchecked backpressure, or downstream bottlenecks, or this one workload ends up consuming an entire worker node, or ... [0] https://www.openshift.com/blog/full-cluster-part-2-protecting-nodes https://www.openshift.com/blog/full-cluster-part-2-protectin...
dilyevsky 6y ago
Limits aren’t consulted for scheduling (except if you have cpu manager enabled on the node it can assign dedicated cores) so the above poster is wrong

tpxl 6y ago

>How does removing limits relate to horizontal vs vertical? Vertical -> give more resources to the program Horizontal -> run more instances of the program Removing limits gives your pods more resources (scaling them vertically) whereas creating more pods creates more copies (scaling horizontally). Assuming parent meant scaling by whole units with "scale atomically", that is you have one or two running programs, not "1.5" if you just give it 50% more resources.

oso2k 6y ago
tpxl gets me. :D Even the "scale atomically" part. People seem to have inferred that I believe that Limits are used by the Scheduler. I don't. But if we set "Request = Limits", we're guaranteeing to the Scheduler that our pod workload will never need more than what is Requested, or, we scale up to a new pod. It seems to me latency is a symptom of the actual issue, not the actual problem. If a workload idles at 25% of Request, 12.5% of Limits (as in TFA), and peaks at 50% of Request, 25% of Limits that seems hugely wasteful. What's more, the workload has several "opportunities" to optimize latency. And uncapping the CPU Limit reduces the latency. If it were me, I'd be asking, "Why does my workload potentially need access to (but not utilization?) 4, 6, 8, 16, 32 cores to reduce its latency?" More often than not, I've been able to help customers reduce their latency by DECREASING the Pod's Requests and Limits, but also INCREASE the replica count (via HPA or manually). It's not a silver bullet, and whether a workload is node.js, JBoss EAP, Spring Boot, or Quarkus does matter to some extent. The first thing I reach for in my k8s toolbox is to scale out. "Many hands make light work" is an old adage. N+1 workloads can usually respond to more traffic than N workloads in a shorter amount of time. k8s' strength is that it is networked and clustered. Forcing one node or a set of nodes to work harder (TFA mentions "isolating" the workload) or vertically scaling is anti-pattern in my book. Especially when you understand the workload pattern well. What is being done here is that nodes (which are likely VMs) are being over-committed [0]. Now, those VMs live on physical hypervisors which are likely -guess what- over-committed. Turtles of (S)POFs all the way down I say. Also, TFA mentions In the past we’ve seen some nodes going to a "notReady" state, mainly because some services were using too much resources in a node. and The downsides are that we lose in “container density”, the number of containers that can run in a single node. We could also end up with a lot of “slack” during a low traffic time. You could also hit some high CPU usage, but nodes autoscaling should help you with it. So they acknowledge the risk is real and they've encountered it. For most of my customers, failing nodes, reduced "container density", and "slack" are unacceptable. That translates into increased engineer troubleshooting time, higher cloud provider bills. What's worse is that the suggestion of the Cluster Autoscaler will protect you also comes with increased costs (licenses, VM, storage, etc.). Not the solution I want. Seems like a blank check to your cloud provider. But I get it. I've fought with customers that tell me, "By removing the Limit, my container starts up in half the time." Great. Then they get to Perf Testing and they get wildly inconsistent speed up when scaling out (or way sublinear), or they're limited by resource in their ability to scale up especially when metrics tells them they have resources available, or there is unchecked backpressure, or downstream bottlenecks, or this one workload ends up consuming an entire worker node, or ... [0] https://www.openshift.com/blog/full-cluster-part-2-protecting-nodes https://www.openshift.com/blog/full-cluster-part-2-protectin...

dilyevsky 6y ago

Limits aren’t consulted for scheduling (except if you have cpu manager enabled on the node it can assign dedicated cores) so the above poster is wrong