4 ms·

I'm curious about your practises regarding CPU Limits at Stripe. Do you noticed severe CPU throttling? What's your guidelines on this?

jmillikin 6y ago
Product teams deploying code to our Kubernetes clusters are strongly recommended to use resource limits, and we're going to make that a hard requirement at some point. We haven't noticed unusual CPU throttling, though we do have some workloads that turned out to be burstier than expected and had to adjust their CPU limits to match. Note that when it comes to subtle Linux thread scheduling behavior, your experience will depend on which runtime you use, and if using runc then which version of the Linux kernel your workers run. We weren't affected by the CFS bug introduced in Linux v4.18 because we never ran Kubernetes workloads on a machine with the affected kernel, and if a similar bug occurs in the future it might not affect workloads running within gVisor or Firecracker. Additionally, Stripe has historically cared more about security than efficiency. This lead to an architecture where services run on dedicated VMs, which naturally strands capacity and reduces the impact of bugs that appear at high utilization and/or high core count.
- xyzzy_plugh 6y ago
  I believe you did run into the CFS bug in non-Kubernetes workloads, though, specifically with Hadoop tasks. An engineer told me about a workaround he devised using cpuset.
  - jmillikin 6y ago
    That's likely a different bug. From what I understand the CFS bug being discussed was introduced in Linux v4.18 and fixed in v5.3, and we have not used a kernel within in that range in our Hadoop clusters.
mnahkies 6y ago
Not from stripe, but I've seen pretty bad CPU throttling. Often see quota's get exhausted through short bursts that don't show up in metrics that then causes CFS throttling to occur even though it looks like the pod is no where near its limit. Also struggled with application startup requiring far more CPU than at runtime leading to ridiculously slow startup times if you had a low limit. So far our solution has been to just remove CPU limits, but hoping things will get better. Removing the limits really improved our latency tail, and so far hasn't resulted in CPU saturation at the node level but your mileage may vary
- tbrock 6y ago
  Did you know that the Linux kernel has a bug that makes CPU limits for containers extra costly? https://github.com/kubernetes/kubernetes/issues/67577 https://github.com/kubernetes/kubernetes/issues/67577 https://github.com/torvalds/linux/commit/512ac999d2755d2b7109e996a76b6fb8b888631d https://github.com/torvalds/linux/commit/512ac999d2755d2b710... If I recall correctly you need 4.18+ to get the fix.
- dilyevsky 6y ago
  We’ve seen just regular throttling too especially with erlang vm and go workloads

jmillikin 6y ago

Product teams deploying code to our Kubernetes clusters are strongly recommended to use resource limits, and we're going to make that a hard requirement at some point. We haven't noticed unusual CPU throttling, though we do have some workloads that turned out to be burstier than expected and had to adjust their CPU limits to match. Note that when it comes to subtle Linux thread scheduling behavior, your experience will depend on which runtime you use, and if using runc then which version of the Linux kernel your workers run. We weren't affected by the CFS bug introduced in Linux v4.18 because we never ran Kubernetes workloads on a machine with the affected kernel, and if a similar bug occurs in the future it might not affect workloads running within gVisor or Firecracker. Additionally, Stripe has historically cared more about security than efficiency. This lead to an architecture where services run on dedicated VMs, which naturally strands capacity and reduces the impact of bugs that appear at high utilization and/or high core count.

xyzzy_plugh 6y ago
I believe you did run into the CFS bug in non-Kubernetes workloads, though, specifically with Hadoop tasks. An engineer told me about a workaround he devised using cpuset.
- jmillikin 6y ago
  That's likely a different bug. From what I understand the CFS bug being discussed was introduced in Linux v4.18 and fixed in v5.3, and we have not used a kernel within in that range in our Hadoop clusters.

mnahkies 6y ago

Not from stripe, but I've seen pretty bad CPU throttling. Often see quota's get exhausted through short bursts that don't show up in metrics that then causes CFS throttling to occur even though it looks like the pod is no where near its limit. Also struggled with application startup requiring far more CPU than at runtime leading to ridiculously slow startup times if you had a low limit. So far our solution has been to just remove CPU limits, but hoping things will get better. Removing the limits really improved our latency tail, and so far hasn't resulted in CPU saturation at the node level but your mileage may vary

tbrock 6y ago
Did you know that the Linux kernel has a bug that makes CPU limits for containers extra costly? https://github.com/kubernetes/kubernetes/issues/67577 https://github.com/kubernetes/kubernetes/issues/67577 https://github.com/torvalds/linux/commit/512ac999d2755d2b7109e996a76b6fb8b888631d https://github.com/torvalds/linux/commit/512ac999d2755d2b710... If I recall correctly you need 4.18+ to get the fix.
dilyevsky 6y ago
We’ve seen just regular throttling too especially with erlang vm and go workloads