6 ms·

The common problem I see across many languages is: applications detect machine cores by looking at /proc/cpuinfo. However, in a docker container (or other cont

by dekhn 3y ago

The common problem I see across many languages is: applications detect machine cores by looking at /proc/cpuinfo. However, in a docker container (or other container technology), that file looks the same as the container host (listing all cores, regardless of how few have been assigned to the container). I wondered for a while if docker could make a fake /proc/cpuinfo that apps could parse that just listed "docker cpus" allocated to the job, but upon further reflection, that probably wouldn't work for many reasons.

dharmab 3y ago
Point of clarification: Containers, when using quota based limits, can use all of the CPU cores on the host. They're limited in how much time they can spend using them. (There are exceptions, such as documented here: https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/ https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana...)
- dekhn 3y ago
  Maybe I should be clearer: Let's say I have a 16 core host and I start a flask container with cpu=0.5 that forks and has a heavy post-fork initializer. flask/gunicorn will fork 16 processes (by reading /proc/cpuinfo and counting cores) all of which will try to share 0.5 cores worth of CPU power (maybe spread over many physical CPUs; I don't really care about that). I can solve this by passing a flag to my application; my complaint is more that apps shouldn't consult /proc/cpuinfo, but have another standard interface to ask "what should I set my max parallelism (NOT CONCURRENCY, ROB) so my worker threads get adequate CPU time so the framework doesn't time out on startup.
  - status_quo69 3y ago
    https://stackoverflow.com/questions/65551215/get-docker-cpu-memory-limit-inside-container https://stackoverflow.com/questions/65551215/get-docker-cpu-... Been a bit but I do believe that dotnet does this exact behavior. Sounds like gunicorn needs a pr to mimic, if they want to replicate this. https://github.com/dotnet/runtime/issues/8485 https://github.com/dotnet/runtime/issues/8485
  - Volundr 3y ago
    It's not clear to me what the max parallelism should actually be on a container with a CPU limit of .5. To my understanding that limits CPU time the container can use within a certain time interval, but doesn't actually limit the parallel processes an application can run. In other words that container with .5 on the CPU limit can indeed use all 16 physical cores of that machine. It'll just burn through it's budget 16x faster. If that's desirable vs limiting itself to one process is going to be highly application dependent and not something kubernetes and docker can just tell you.
    - FridgeSeal 3y ago
      It won’t burn through the budget faster by having more cores. You’re given a fixed time-slice of the whole CPU (in K8s, caveats below), whether you use all the cores or just one doesn’t particularly matter. On one hand, it would be would nice to be able to limit workloads on K8s to a subset of cores too, on the other, I can only imagine how much catastrophically complex that would make scheduling and optimisation. Caveats: up to the number of cores exposed to your VM. I also believe the later versions of K8s let you do some degree of workload-core pinning and I don’t yet know how that interacts with core availability .
  - vbezhenar 3y ago
    You generally shouldn't set CPU limits. You might want to configure CPU requests which is guaranteed chunk of CPU time that container will always receive. With CPU limits you'll encounter situation when host CPU is not loaded, but your container workloaded is throttled at the same time, which is just waste of CPU resources.
    - lovasoa 3y ago
      According to the article, this is not true. The limits become active only when the host cpu is under pressure.
      - vbezhenar 3y ago
        I don't think that's correct. --cpus is the same as --cpu-period which is cpu limit. You can easily check it yourself, just run docker container with --cpus set, run multi-core load there and check your activity monitor.
        pbh101 3y ago
        CFS quotas only become active under contention and even then are relative: if you’re the only thing running on the box and want all the cores but only set one cpu, you get all of them anyway. If you set cpus to 2 and another process sets to 1 and you both try to use all CPUs all out, you’ll get 66% and they’ll get 33%. This isn’t the same as cpusets, which work differently.
        ecnahc515 3y ago
        > CFS quotas only become active under contention That's not true at all. Take a look at `cpu.cfs_quota_us` in https://kernel.googlesource.com/pub/scm/linux/kernel/git/glommer/memcg/+/cpu_stat/Documentation/cgroups/cpu.txt https://kernel.googlesource.com/pub/scm/linux/kernel/git/glo... It's a hard time limit. It doesn't care about contention at all. `cpu.shares` is relative, for choosing which process gets scheduled, and how often, but the CFS quota is a hard limit on runtime.
  - wutwutwat 3y ago
    `gunicorn --workers $(nproc)`, see my comment on the parent
  - dharmab 3y ago
    That interface partly exists. It's /sys/fs/cgroup/(cgroup here)/cpu.max I know the JVM automatically uses it, and there's a popular library for Go that sets GONAXPROCS using it.
jeffbee 3y ago
That's not what Go does though. Go looks at the population of the CPU mask at startup. It never looks again, which of problematic in K8s where the visible CPUs may change while your process runs.
- dekhn 3y ago
  What is the population of the CPU mask at startup? Is this a kernel call? A /proc file? Some register?
  - EdSchouten 3y ago
    On Linux, it likely calls sched_getaffinity().
    - dekhn 3y ago
      hmm, I can see that as being useful but I also don't see that as the way to determine "how many worker threads I should start"
      - jeffbee 3y ago
        It's not a bad way to guess, up to maybe 16 or so. Most Go server programs aren't going to just scale up forever, so having 188 threads might be a waste. Just setting it to 16 will satisfy 99% of users.
        dekhn 3y ago
        There's going to be a bunch of missing info, though, in some cases I can think of. For example, more and more systems have asymmetric cores. /proc/cpuinfo can expose that information in detail, including (current) clock speed, processor type, etc, while cpu_set is literally just a bitmask (if I read the man pages right) of system cores your process is allowed to schedule on. Fundamentally, intelligent apps need to interrogate their environment to make concurrency decisions. But I agree- Go would probably work best if it just picked a standard parallelism constant like 16 and just let users know that can be tuned if they have additional context.
wutwutwat 3y ago
I only use `nproc` and see it used in other containers as well, ie `bundle install -j $(nproc)`. This honors cpu assignment and provides the functionality you're seeking. Whether or not random application software uses nproc if available, idk > Print the number of processing units available to the current process, which may be less than the number of online processors. If this information is not accessible, then print the number of processors installed https://www.gnu.org/software/coreutils/manual/html_node/nproc-invocation.html https://www.gnu.org/software/coreutils/manual/html_node/npro... https://www.flamingspork.com/blog/2020/11/25/why-you-should-use-nproc-and-not-grep-proc-cpuinfo/ https://www.flamingspork.com/blog/2020/11/25/why-you-should-...
- telotortium 3y ago
  This is not very robust. You probably should use the cgroup cpu limits where present, since `docker --cpus` uses a different way to set quota: if [[ -e /sys/fs/cgroup/cpu/cpu.cfs_quota_us ]] && [[ -e /sys/fs/cgroup/cpu/cpu.cfs_period_us ]]; then GOMAXPROCS=$(perl -e 'use POSIX; printf "%d\n", ceil($ARGV[0] / $ARGV[1])' "$(cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us)" "$(cat /sys/fs/cgroup/cpu/cpu.cfs_period_us)") else GOMAXPROCS=$(nproc) fi export GOMAXPROCS This follows from how `docker --cpus` works (https://docs.docker.com/config/containers/resource_constraints/ https://docs.docker.com/config/containers/resource_constrain...), as well as https://stackoverflow.com/a/65554131/207384 https://stackoverflow.com/a/65554131/207384 to get the /sys paths to read from. Or use https://github.com/uber-go/automaxprocs https://github.com/uber-go/automaxprocs, which is very comprehensive, but is a bunch of code for what should be a simple task.
  - fn-mote 3y ago
    A shell script that invokes perl to set an environment variable used by Go. Some days I feel like there is a lot of duct tape involved in these applications.
  - crabbone 3y ago
    Yeah, I'd also go with cgroups... but... You need to know if it's cgroups v1 or v2, where this filesystem is mounted, how to find your own process. Also, there's this hierarchical stuff going on there... also this can change dynamically while your program is running...
rrdharan 3y ago
Containers are a crappy abstraction and VMware fumbled the bag, is my takeaway from this comment…
- sofixa 3y ago
  > VMware fumbled the bag Oh they did, they're a modern day IBM. > Containers are a crappy abstraction They're one of the best abstractions we have (so far) because they contain only the application and what it needs.
  - crabbone 3y ago
    > they contain only the application and what it needs. Delusion level: over 9900. I'm yet to find a container that contains only the application and what it needs. Most of the time I find that they contain at least libc and libpthreads (which are already present on the host, so not needed). More often I find metric tonnes of garbage that was not necessary by any metric, but was just too hard to remove, so was allowed to stay.
    - demi56 3y ago
      Blame that to glibc not the container
irvinlim 3y ago
> I wondered for a while if docker could make a fake /proc/cpuinfo This exists: https://github.com/lxc/lxcfs https://github.com/lxc/lxcfs lxcfs is a FUSE filesystem that mocks /proc by inferring cgroup values in a way that makes other applications and libraries work without having to care about whether it runs in a container (to the best of its ability - there are definitely caveats). One such example is that /proc/uptime should reflect the uptime of the container, not the host; additionally /proc/cpuinfo reflects the number of CPUs as a combination of cpu.max and cpuset.cpus (whichever the lower bound is). As others also mentioned, inferring the number of CPUs could also be done using the sched_getaffinity syscall - this doesn't depend on /proc/cpuinfo, so depending on the library you're using you might be in a pickle.

by dekhn 3y ago

dharmab 3y ago

Point of clarification: Containers, when using quota based limits, can use all of the CPU cores on the host. They're limited in how much time they can spend using them. (There are exceptions, such as documented here: https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/ https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana...)

dekhn 3y ago
Maybe I should be clearer: Let's say I have a 16 core host and I start a flask container with cpu=0.5 that forks and has a heavy post-fork initializer. flask/gunicorn will fork 16 processes (by reading /proc/cpuinfo and counting cores) all of which will try to share 0.5 cores worth of CPU power (maybe spread over many physical CPUs; I don't really care about that). I can solve this by passing a flag to my application; my complaint is more that apps shouldn't consult /proc/cpuinfo, but have another standard interface to ask "what should I set my max parallelism (NOT CONCURRENCY, ROB) so my worker threads get adequate CPU time so the framework doesn't time out on startup.
- status_quo69 3y ago
  https://stackoverflow.com/questions/65551215/get-docker-cpu-memory-limit-inside-container https://stackoverflow.com/questions/65551215/get-docker-cpu-... Been a bit but I do believe that dotnet does this exact behavior. Sounds like gunicorn needs a pr to mimic, if they want to replicate this. https://github.com/dotnet/runtime/issues/8485 https://github.com/dotnet/runtime/issues/8485
- Volundr 3y ago
  It's not clear to me what the max parallelism should actually be on a container with a CPU limit of .5. To my understanding that limits CPU time the container can use within a certain time interval, but doesn't actually limit the parallel processes an application can run. In other words that container with .5 on the CPU limit can indeed use all 16 physical cores of that machine. It'll just burn through it's budget 16x faster. If that's desirable vs limiting itself to one process is going to be highly application dependent and not something kubernetes and docker can just tell you.
  - FridgeSeal 3y ago
    It won’t burn through the budget faster by having more cores. You’re given a fixed time-slice of the whole CPU (in K8s, caveats below), whether you use all the cores or just one doesn’t particularly matter. On one hand, it would be would nice to be able to limit workloads on K8s to a subset of cores too, on the other, I can only imagine how much catastrophically complex that would make scheduling and optimisation. Caveats: up to the number of cores exposed to your VM. I also believe the later versions of K8s let you do some degree of workload-core pinning and I don’t yet know how that interacts with core availability .
- vbezhenar 3y ago
  You generally shouldn't set CPU limits. You might want to configure CPU requests which is guaranteed chunk of CPU time that container will always receive. With CPU limits you'll encounter situation when host CPU is not loaded, but your container workloaded is throttled at the same time, which is just waste of CPU resources.
  - lovasoa 3y ago
    According to the article, this is not true. The limits become active only when the host cpu is under pressure.
    - vbezhenar 3y ago
      I don't think that's correct. --cpus is the same as --cpu-period which is cpu limit. You can easily check it yourself, just run docker container with --cpus set, run multi-core load there and check your activity monitor.
      - pbh101 3y ago
        CFS quotas only become active under contention and even then are relative: if you’re the only thing running on the box and want all the cores but only set one cpu, you get all of them anyway. If you set cpus to 2 and another process sets to 1 and you both try to use all CPUs all out, you’ll get 66% and they’ll get 33%. This isn’t the same as cpusets, which work differently.
        ecnahc515 3y ago
        > CFS quotas only become active under contention That's not true at all. Take a look at `cpu.cfs_quota_us` in https://kernel.googlesource.com/pub/scm/linux/kernel/git/glommer/memcg/+/cpu_stat/Documentation/cgroups/cpu.txt https://kernel.googlesource.com/pub/scm/linux/kernel/git/glo... It's a hard time limit. It doesn't care about contention at all. `cpu.shares` is relative, for choosing which process gets scheduled, and how often, but the CFS quota is a hard limit on runtime.
- wutwutwat 3y ago
  `gunicorn --workers $(nproc)`, see my comment on the parent
- dharmab 3y ago
  That interface partly exists. It's /sys/fs/cgroup/(cgroup here)/cpu.max I know the JVM automatically uses it, and there's a popular library for Go that sets GONAXPROCS using it.

jeffbee 3y ago

That's not what Go does though. Go looks at the population of the CPU mask at startup. It never looks again, which of problematic in K8s where the visible CPUs may change while your process runs.

dekhn 3y ago
What is the population of the CPU mask at startup? Is this a kernel call? A /proc file? Some register?
- EdSchouten 3y ago
  On Linux, it likely calls sched_getaffinity().
  - dekhn 3y ago
    hmm, I can see that as being useful but I also don't see that as the way to determine "how many worker threads I should start"
    - jeffbee 3y ago
      It's not a bad way to guess, up to maybe 16 or so. Most Go server programs aren't going to just scale up forever, so having 188 threads might be a waste. Just setting it to 16 will satisfy 99% of users.
      - dekhn 3y ago
        There's going to be a bunch of missing info, though, in some cases I can think of. For example, more and more systems have asymmetric cores. /proc/cpuinfo can expose that information in detail, including (current) clock speed, processor type, etc, while cpu_set is literally just a bitmask (if I read the man pages right) of system cores your process is allowed to schedule on. Fundamentally, intelligent apps need to interrogate their environment to make concurrency decisions. But I agree- Go would probably work best if it just picked a standard parallelism constant like 16 and just let users know that can be tuned if they have additional context.

wutwutwat 3y ago

I only use `nproc` and see it used in other containers as well, ie `bundle install -j $(nproc)`. This honors cpu assignment and provides the functionality you're seeking. Whether or not random application software uses nproc if available, idk > Print the number of processing units available to the current process, which may be less than the number of online processors. If this information is not accessible, then print the number of processors installed https://www.gnu.org/software/coreutils/manual/html_node/nproc-invocation.html https://www.gnu.org/software/coreutils/manual/html_node/npro... https://www.flamingspork.com/blog/2020/11/25/why-you-should-use-nproc-and-not-grep-proc-cpuinfo/ https://www.flamingspork.com/blog/2020/11/25/why-you-should-...

telotortium 3y ago
This is not very robust. You probably should use the cgroup cpu limits where present, since `docker --cpus` uses a different way to set quota: if [[ -e /sys/fs/cgroup/cpu/cpu.cfs_quota_us ]] && [[ -e /sys/fs/cgroup/cpu/cpu.cfs_period_us ]]; then GOMAXPROCS=$(perl -e 'use POSIX; printf "%d\n", ceil($ARGV[0] / $ARGV[1])' "$(cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us)" "$(cat /sys/fs/cgroup/cpu/cpu.cfs_period_us)") else GOMAXPROCS=$(nproc) fi export GOMAXPROCS This follows from how `docker --cpus` works (https://docs.docker.com/config/containers/resource_constraints/ https://docs.docker.com/config/containers/resource_constrain...), as well as https://stackoverflow.com/a/65554131/207384 https://stackoverflow.com/a/65554131/207384 to get the /sys paths to read from. Or use https://github.com/uber-go/automaxprocs https://github.com/uber-go/automaxprocs, which is very comprehensive, but is a bunch of code for what should be a simple task.
- fn-mote 3y ago
  A shell script that invokes perl to set an environment variable used by Go. Some days I feel like there is a lot of duct tape involved in these applications.
- crabbone 3y ago
  Yeah, I'd also go with cgroups... but... You need to know if it's cgroups v1 or v2, where this filesystem is mounted, how to find your own process. Also, there's this hierarchical stuff going on there... also this can change dynamically while your program is running...

rrdharan 3y ago

Containers are a crappy abstraction and VMware fumbled the bag, is my takeaway from this comment…

sofixa 3y ago
> VMware fumbled the bag Oh they did, they're a modern day IBM. > Containers are a crappy abstraction They're one of the best abstractions we have (so far) because they contain only the application and what it needs.
- crabbone 3y ago
  > they contain only the application and what it needs. Delusion level: over 9900. I'm yet to find a container that contains only the application and what it needs. Most of the time I find that they contain at least libc and libpthreads (which are already present on the host, so not needed). More often I find metric tonnes of garbage that was not necessary by any metric, but was just too hard to remove, so was allowed to stay.
  - demi56 3y ago
    Blame that to glibc not the container

irvinlim 3y ago

> I wondered for a while if docker could make a fake /proc/cpuinfo This exists: https://github.com/lxc/lxcfs https://github.com/lxc/lxcfs lxcfs is a FUSE filesystem that mocks /proc by inferring cgroup values in a way that makes other applications and libraries work without having to care about whether it runs in a container (to the best of its ability - there are definitely caveats). One such example is that /proc/uptime should reflect the uptime of the container, not the host; additionally /proc/cpuinfo reflects the number of CPUs as a combination of cpu.max and cpuset.cpus (whichever the lower bound is). As others also mentioned, inferring the number of CPUs could also be done using the sched_getaffinity syscall - this doesn't depend on /proc/cpuinfo, so depending on the library you're using you might be in a pickle.