3 ms·

Even this I feel doesn't work all the time. At least from memory, I remember some problems where on kubernetes if you look at nproc or cpuinfo, that will give

by cbanek 6y ago

Even this I feel doesn't work all the time. At least from memory, I remember some problems where on kubernetes if you look at nproc or cpuinfo, that will give you the number of processors on the host machine, and not necessarily the CPU limit which is set per container. Although maybe I'm mis-remembering. I guess nproc could look at something else as well to filter based on that, and I agree having a nice portable commandline is way better than looking at /proc.

lathiat 6y ago
This is 100% a problem I was working on for a customer recently. nproc specifically does not currently check your CFS CPU cgroup quota, i.e. you may have 12 active cores but only a budget of 4 CPUs worth of CPU time. This can be an issue. It would be ideal to have a standard utility or library that would also check that. If you want to understand more about these problems this article has some good background, although they fixed a real bug in the scheduler here the background will give you the general idea and it's still a problem to some extent if all of those threads/processes are trying to contend on a single shared lock.. which is precisely what happens with multi threaded Python programs (they sometimes contend on the GIL) that use data science modules which do a lot of processing in C without the GIL lock held and only take it occasionally. However the problem comes if you have 12 threads consuming your 4 threads worth of CPU time they'll end up block for 5-10 or more milliseconds at a time where they may well be holding that lock and preventing the other threads from running. You may then ask why not reduce the CPU set for those containers, the problem is there is no way (at least currently) to say 'this process can only run on 4 CPUs, but any of them'. You have to reduce the CPU set down to 4 specific CPUs and now if all containers scheduled to those specific 4 CPUs are busy you will be losing time to those other containers when a different 4 CPUs may be totally idle. It would be nice to advance the scheduler to support such a concept without using the quotas specifically. As a final side note.. LXD cheats here. It uses 'lxcfs' to bind mount onto /proc/cpuinfo and modifies the output to only show the number of CPU cores you actually have. Which suffices for most programs.
EdSchouten 6y ago
If you just run Kubernetes with the default settings, that’s correct. You will need to enable the so-called static cpu manager policy on kubelets to make it use cpusets based on CPU requests & limits. https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/ https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana... A somewhat annoying thing about this option is that even though it can create cpusets for every container individually, it only gets enabled in case all containers in the pod use an integer number of CPUs, which seems unnecessary.

by cbanek 6y ago

lathiat 6y ago

This is 100% a problem I was working on for a customer recently. nproc specifically does not currently check your CFS CPU cgroup quota, i.e. you may have 12 active cores but only a budget of 4 CPUs worth of CPU time. This can be an issue. It would be ideal to have a standard utility or library that would also check that. If you want to understand more about these problems this article has some good background, although they fixed a real bug in the scheduler here the background will give you the general idea and it's still a problem to some extent if all of those threads/processes are trying to contend on a single shared lock.. which is precisely what happens with multi threaded Python programs (they sometimes contend on the GIL) that use data science modules which do a lot of processing in C without the GIL lock held and only take it occasionally. However the problem comes if you have 12 threads consuming your 4 threads worth of CPU time they'll end up block for 5-10 or more milliseconds at a time where they may well be holding that lock and preventing the other threads from running. You may then ask why not reduce the CPU set for those containers, the problem is there is no way (at least currently) to say 'this process can only run on 4 CPUs, but any of them'. You have to reduce the CPU set down to 4 specific CPUs and now if all containers scheduled to those specific 4 CPUs are busy you will be losing time to those other containers when a different 4 CPUs may be totally idle. It would be nice to advance the scheduler to support such a concept without using the quotas specifically. As a final side note.. LXD cheats here. It uses 'lxcfs' to bind mount onto /proc/cpuinfo and modifies the output to only show the number of CPU cores you actually have. Which suffices for most programs.

EdSchouten 6y ago

If you just run Kubernetes with the default settings, that’s correct. You will need to enable the so-called static cpu manager policy on kubelets to make it use cpusets based on CPU requests & limits. https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/ https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana... A somewhat annoying thing about this option is that even though it can create cpusets for every container individually, it only gets enabled in case all containers in the pod use an integer number of CPUs, which seems unnecessary.