4 ms·

What's the accurate way?

by nulltype 11y ago

uxcn 11y ago
To calculate the number of processors available or the optimal number of threads/processes?
- nulltype 11y ago
  Well specifically it would be cool if it counted the optimal number to set GOMAXPROCS to, since I think that's like the main use of runtime.NumCPU().
  - thrownaway2424 11y ago
    Right. Currently runtime.NumCPU tries to be fancy by looking at the population count of the cpuset mask[1]. However in a hosted environment using containers there's no reason to believe that the cpuset will remain fixed over the life of the process. This can undercount the available CPUs, leaving you with a GOMAXPROCS that is too low. 1: https://code.google.com/p/go/source/browse/src/pkg/runtime/thread_linux.c?r=ef1158a7371796bf4823a1ce43e3d01d2a765e14#85 https://code.google.com/p/go/source/browse/src/pkg/runtime/t...
    - vessenes 11y ago
      Anecdotally, it's very often not bad, (and in fact sometimes "good") to over-provision MAXPROCS. We have used as much as 3 to 6x the number of hyperthreaded cores with good results, depending on the workload. This could insulate you against some container changes.
    - nulltype 11y ago
      Seems like if they did some other metric, it could overcount the number of CPUs instead, right? What about changing GOMAXPROCS once per minute in a goroutine that calls NumCPU()?
dunkelheit 11y ago
At the very least you should be aware that these counts usually count cores with hyperthreading twice - and hyperthreading does indeed provide opportunity for increased parallelism but it is noticeably worse than having another separate physical core.
wmf 11y ago
Probably something like hwloc which can be configured to show you only the cpus you're allowed to use. BTW the same problem also happens for determining available memory in containers.
seiji 11y ago
As usual, erlang does it better: http://www.erlang.org/doc/man/erlang.html#system_info_cpu_topology http://www.erlang.org/doc/man/erlang.html#system_info_cpu_to... and http://www.erlang.org/doc/man/erlang.html#logical_processors http://www.erlang.org/doc/man/erlang.html#logical_processors

What's the accurate way?

uxcn 11y ago

To calculate the number of processors available or the optimal number of threads/processes?

nulltype 11y ago
Well specifically it would be cool if it counted the optimal number to set GOMAXPROCS to, since I think that's like the main use of runtime.NumCPU().
- thrownaway2424 11y ago
  Right. Currently runtime.NumCPU tries to be fancy by looking at the population count of the cpuset mask[1]. However in a hosted environment using containers there's no reason to believe that the cpuset will remain fixed over the life of the process. This can undercount the available CPUs, leaving you with a GOMAXPROCS that is too low. 1: https://code.google.com/p/go/source/browse/src/pkg/runtime/thread_linux.c?r=ef1158a7371796bf4823a1ce43e3d01d2a765e14#85 https://code.google.com/p/go/source/browse/src/pkg/runtime/t...
  - vessenes 11y ago
    Anecdotally, it's very often not bad, (and in fact sometimes "good") to over-provision MAXPROCS. We have used as much as 3 to 6x the number of hyperthreaded cores with good results, depending on the workload. This could insulate you against some container changes.
  - nulltype 11y ago
    Seems like if they did some other metric, it could overcount the number of CPUs instead, right? What about changing GOMAXPROCS once per minute in a goroutine that calls NumCPU()?

dunkelheit 11y ago

At the very least you should be aware that these counts usually count cores with hyperthreading twice - and hyperthreading does indeed provide opportunity for increased parallelism but it is noticeably worse than having another separate physical core.

wmf 11y ago

Probably something like hwloc which can be configured to show you only the cpus you're allowed to use. BTW the same problem also happens for determining available memory in containers.

seiji 11y ago

As usual, erlang does it better: http://www.erlang.org/doc/man/erlang.html#system_info_cpu_topology http://www.erlang.org/doc/man/erlang.html#system_info_cpu_to... and http://www.erlang.org/doc/man/erlang.html#logical_processors http://www.erlang.org/doc/man/erlang.html#logical_processors