4 ms·

This is amusing next to "Go code that uses 20% of CPU time in GC is usually bad Go code," but how much does it matter? Kubelet runs on every machine in the clus

This is amusing next to "Go code that uses 20% of CPU time in GC is usually bad Go code," but how much does it matter? Kubelet runs on every machine in the cluster, but I'd expect few of each machine's cycles to be spent on it. I wouldn't be surprised if relatively little attention has been paid to optimizing its GC and/or if someone has deliberately tuned it to reduce RAM at the expense of GC CPU. Put another way: this percentage uses the wrong denominator for ranking optimization targets. Don't rank by percentage of the binary's cycles but instead by percentage of overall cluster cycles. This is particularly true for Kubernetes itself. It affects the efficiency of the cluster as a whole via its bin-packing decisions. Eg the whole cluster becomes more efficient when Kubernetes minimizes stranded capacity and also when it isolates workloads that heavily thrash the CPU cache to separate NUMA nodes or machines. Thus if I were to dive into Kubernetes optimization, I'd focus on bin-packing much more than its own GC cycles.

ltbarcly3 5y ago
I don't think the point is that this is causing wasted CPU cycles, it's that if something shows symptoms of being written by incompetent programmers, it's likely they were incompetent in many other ways.
- scottlamb 5y ago
  That's what makes it amusing, but I don't necessarily agree with "Go code that uses 20% of CPU time in GC is usually bad Go code", and I certainly would disagree if you omitted the word "usually". Quality code is optimized based on what really matters, and I'm skeptical GC cycles really matter here.
  - smarterclayton 5y ago
    Right now it’s mostly the CRI/kubelet interaction (which polls the container runtime for containers) and metrics scraping generating the garbage. Having an incremental CRI watch (vs poll) has been on the backlog for years, but correctness and features have been more important for most people. Several people were looking at this recently as we were squeezing management pods + system processes onto two dedicated cores for telco/edge real-time use cases (the other tens of cores being reserved for workloads). It’s not that hard to fix, it’s just never been the top priority. Real software has curves.

ltbarcly3 5y ago

I don't think the point is that this is causing wasted CPU cycles, it's that if something shows symptoms of being written by incompetent programmers, it's likely they were incompetent in many other ways.

scottlamb 5y ago
That's what makes it amusing, but I don't necessarily agree with "Go code that uses 20% of CPU time in GC is usually bad Go code", and I certainly would disagree if you omitted the word "usually". Quality code is optimized based on what really matters, and I'm skeptical GC cycles really matter here.
- smarterclayton 5y ago
  Right now it’s mostly the CRI/kubelet interaction (which polls the container runtime for containers) and metrics scraping generating the garbage. Having an incremental CRI watch (vs poll) has been on the backlog for years, but correctness and features have been more important for most people. Several people were looking at this recently as we were squeezing management pods + system processes onto two dedicated cores for telco/edge real-time use cases (the other tens of cores being reserved for workloads). It’s not that hard to fix, it’s just never been the top priority. Real software has curves.