4 ms·
Use `nproc` and not grep /proc/cpuinfo
- 2bluesc 6y agoFor those running `make`, add this to you shell init to set the concurrency level via env var: > expert MAKEFLAGS=-j$(nproc)
- smabie 6y ago*export
- s_gourichon 6y agoGood idea. make -j without a number should do that, instead of spawning processes like there’s no tomorrow.
- moonchild 6y agoTo be fair, having some way to spawn processes like there's no tomorrow is a good way to make sure there are no concurrency issues in your makefile. I had a makefile with a race condition that triggered only sporadically. However, giving it more threads significantly increased its rate of occurrence.
- PikachuEXE 6y agoAny doc that suggest this behaviour?
- exciteabletom 6y agoFrom the man page: > If the -j option is given without an argument, make will not limit the number of jobs that can run simultaneously.
- stewartesmith 6y agoWhat this practically means is that a plain "make -j" will spawn all the processes that could run concurrently. So if you're building 1,000 files, it'll spawn 1,000 processes. This can either be fine, or completely terrible, depending on the number of CPUs you have an the amount of memory each process uses.
- moonchild 6y agomake -j$(($(nproc)+2))
- _flux 6y agoAs a software developer one might also choose to use the switch -O as to not mangle the output in a (too) nondeterministic order. Or prefer ninja overall.
- fillest 6y agoSome software doesn't support that - building that way can randomly fail with very confusing errors.
- account42 6y agoThen that software should add the proper dependencies to their makefile.
- s_gourichon 6y agoInteresting, but how does nproc get the right information ? OMP_NUM_THREADS and OMP_THREAD_LIMIT are in principle extra limitations, not the main source of information, right ?
- formerly_proven 6y agoIt's sched_getaffinity, which makes sense, since that's what taskset and friends modify. $ strace nproc sched_getaffinity(0, 128, [0, 1, 2, ..., 78, 79]) = 40 fstat(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0xf), ...}) = 0 write(1, "80\n", 324) = 3 exit_group(0) = ? +++ exited with 0 +++
- nemetroid 6y agoIf you have OPM_NUM_THREADS and OMP_THREAD_LIMIT set it will use those, though.
- shmerl 6y agoYeah, I just use $(nproc) in scripts when I need to set up threads number.
- skizziepop 6y agonproc isn't available on some of the BSDs
- oso2k 6y agoRight. So on Mac or other BSDs, you want use a ‘sysctl’.
- oso2k 6y agoAnother thought, I don’t know if Alpine Linux or other musl-libc based systems will have ‘nproc’ if, for instance, you’re running containers or other minimal systems.
- sigjuice 6y agohttps://github.com/mirror/busybox/blob/master/coreutils/nproc.c https://github.com/mirror/busybox/blob/master/coreutils/npro...
- floatingatoll 6y agoOn Macs, this "total cores available" strategy might not be a valid approach, since you probably only want to be using the high-performance cores and not also the low-power ones — and the CPU might well overbook all your threads to high-perf to leave room for the OS and other programs, even if you try to do otherwise.
- stewartesmith 6y agoNot just Macs, plenty of Linux systems have a big.LITTLE design. with a big.LITTLE system it'll really depend what you're doing. Just compiling something? It's probably fine (if all cores can be active at once). If you're wanting more consistent performance for some calculations? You might need to think a bit more.
- toast0 6y agocpuset -g on FreeBSD >= 7.1 will tell you what cpus are available to your shell, where a sysctl will only tell you what's on your system. (Lots of good info in the related manual pages)
- oso2k 6y agoI see suggestions to use ‘make -j$(nproc)’. However, I was taught when learning MPI programming 20 years ago, you want think at least “one to load, one to run”. The idea being that processes and threads waiting on IO will sleep/yield while another process or thread can continue executing. An especially useful strategy if you can afford the RAM (trading RAM for CPU). Consequently, I tend to run twice as many ‘make’ jobs as threads on my systems.
- richardwhiuk 6y agoI think that would justify adding a single extra, not an entire slate.
- oso2k 6y agoI think logic still holds. There’s an order magnitude more cache in today’s server CPUs. There’s higher likelihood today that your thread or process will remain in the cache, keeping it performant.
- edgyquant 6y agoYeah I think so I was taught to run make -j(number of threads + 1) by taught I mean I saw it in a lot of tutorials during my early years of learning Linux, C and make.
- kevin_thibedeau 6y agoThat was before mainstream SMT.
- flafla2 6y agoI’ve heard about this trick/incantation as well, but haven’t managed to measure a significant performance gain (though it makes sense in theory). Do you know of any quantitative evidence that this is solid advice? I suppose my testing is not at all scientific, just my anecdotal experience using make -j(i+1)
- usefulcat 6y agoThis is probably more worthwhile when using spinning disks, as opposed to SSDs.
- swills 6y agoThe portable way is: getconf _NPROCESSORS_ONLN
- slrz 6y agoThis is asking for a different thing, though. % taskset 1 nproc 1 % taskset 1 getconf _NPROCESSORS_ONLN 8
- cycloptic 6y agoIIRC this utility is a wrapper around glibc get_nprocs(), which on Linux will ultimately read from /sys/devices/system/cpu/online. If you need this information from a compiled language, you should probably use the glibc wrapper.
- rwmj 6y agoWe usually use np = sysconf(_SC_NPROCESSORS_ONLN); However I'm interested in whether this is cgroups- (hence container-) aware or not. I'm not actually sure, and it will affect some of my software when running in a container.
- cbanek 6y agoEven this I feel doesn't work all the time. At least from memory, I remember some problems where on kubernetes if you look at nproc or cpuinfo, that will give you the number of processors on the host machine, and not necessarily the CPU limit which is set per container. Although maybe I'm mis-remembering. I guess nproc could look at something else as well to filter based on that, and I agree having a nice portable commandline is way better than looking at /proc.
- lathiat 6y agoThis is 100% a problem I was working on for a customer recently. nproc specifically does not currently check your CFS CPU cgroup quota, i.e. you may have 12 active cores but only a budget of 4 CPUs worth of CPU time. This can be an issue. It would be ideal to have a standard utility or library that would also check that. If you want to understand more about these problems this article has some good background, although they fixed a real bug in the scheduler here the background will give you the general idea and it's still a problem to some extent if all of those threads/processes are trying to contend on a single shared lock.. which is precisely what happens with multi threaded Python programs (they sometimes contend on the GIL) that use data science modules which do a lot of processing in C without the GIL lock held and only take it occasionally. However the problem comes if you have 12 threads consuming your 4 threads worth of CPU time they'll end up block for 5-10 or more milliseconds at a time where they may well be holding that lock and preventing the other threads from running. You may then ask why not reduce the CPU set for those containers, the problem is there is no way (at least currently) to say 'this process can only run on 4 CPUs, but any of them'. You have to reduce the CPU set down to 4 specific CPUs and now if all containers scheduled to those specific 4 CPUs are busy you will be losing time to those other containers when a different 4 CPUs may be totally idle. It would be nice to advance the scheduler to support such a concept without using the quotas specifically. As a final side note.. LXD cheats here. It uses 'lxcfs' to bind mount onto /proc/cpuinfo and modifies the output to only show the number of CPU cores you actually have. Which suffices for most programs.
- EdSchouten 6y agoIf you just run Kubernetes with the default settings, that’s correct. You will need to enable the so-called static cpu manager policy on kubelets to make it use cpusets based on CPU requests & limits. https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/ https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana... A somewhat annoying thing about this option is that even though it can create cpusets for every container individually, it only gets enabled in case all containers in the pod use an integer number of CPUs, which seems unnecessary.
- catlifeonmars 6y ago> So, what this will end up doing is just increase the number of context switches, possibly also adding a performance degradation. I think this is a jumping to conclusions a bit. You’d need to be reading /proc/cpuinfo in a fairly tight loop in order for the time spend doing any context switching to dominate the runtime cost of a program. Is this a common practice in some types of computation? Or am I misunderstanding this quote from the article?
- EE84M3i 6y agoThe article is saying if you make too many processes (more than the number of hardware threads available) the OS will have to context switch between them to keep them all running.
- kcbanner 6y agoThey mean that you read it once, then spawn more threads than you have actual cores. Those threads then need to context switch back and forth to do work.
- jasontedor 6y agoI think that the full context of the sentence you quoted will help: > In this case, if you base your number of threads off grepping lscpu you take another dependency (on the util-linux package), which isn’t needed. You also get the wrong answer, as you do by grepping /proc/cpuinfo. So, what this will end up doing is just increase the number of context switches, possibly also adding a performance degradation. The idea is that a common way to size an application that exploits concurrency is to base the number of application threads on the hardware threads available. This portion of the article is saying that if you do so on the basis of `lscpu` or grepping `/proc/cpuinfo` you could mis-size relative to the actual number of cores available to schedule the threads of your application on. Doing so will lead to excessive context switching for your application, and that can hurt.
- stewartesmith 6y agoI wasn't talking about the reading of the number of CPUs you could use, but rather how many threads you'd spawn based on that information. There may be 10 CPUs in the machine, but if your in your container which is restricted to using only 1 of them, it's no use launching 10 threads in that container over just having one.
- cyphar 6y agoIt should be noted that the reason this is wrong under Docker is that /proc/cpuinfo is not cgroup-aware. LXC works around this by creating a fake /proc/cpuinfo (and /proc/meminfo)[1] which matches the values set by cgroup limits. [1]: https://github.com/lxc/lxcfs https://github.com/lxc/lxcfs
- musicale 6y agoOver time more things tend to get containerized/virtualized properly so these sorts of issues can be fixed.
- cyphar 6y agoMy understanding is that it's unlikely cpuinfo and meminfo will be made cgroup-aware due to procfs-specific conventions on magic files. Making global procfs files change based on the current process is seen as a bad design these days -- should it be based on the process reading or the process opening the file (in this instance, both solutions are bad but for different reasons)? From memory, the topic was brought up early on in cgroup development.
- bruce_one 6y agoLXCFS can be used with Docker as well (although the integration is manual) which can be quite helpful (eg we're using it with docker-compose and find it very helpful)
- nomorecodehere8 6y agoIt's amazing how much knowledge is being lost with the advent of container revolutions and orchestrator Os'es. Cgroups is _still_ opt in not opt out...though those who never have worked in HPC on bare metal with parallel libraries and approaches for cpu like openmp are loudly complaining about control groups that don't control or provide consistent view of userland like there isn't a host system underneath. It's like this (https://www.gnu.org/software/libc/manual/html_node/CPU-Affinity.html https://www.gnu.org/software/libc/manual/html_node/CPU-Affin...) wasn't the way things were done (and still are to this day) in everything but the container world. I know it's hard to grasp but not everyone is containerizing everything.