3 ms·

Having 24 threads doesn't mean that -j24 is the sweet spot. I have a machine 56 thread Xenon machine for building git.git, and I find that with its sockets/thre

by avar 7y ago

Having 24 threads doesn't mean that -j24 is the sweet spot. I have a machine 56 thread Xenon machine for building git.git, and I find that with its sockets/threads-per-core/threads-per-socket of 2/2/14 the sweet spot is closer to sockets*threads-per-socket = -j28. Things speed up rather linearly up to -j28, but once I get past -j28 (say -j32) it levels off, and -j56 starts being counterproductive. Same thing with the 160 thread POWER8 machine I have access to. That one runs 8 threads per core, and CPU-limited -jN tops out at around -j20. All of this is very worflow and CPU specific, but generally speaking don't blindly trust what things like "htop" show you as the number of available CPUs, under the hood many of them aren't "real".

gigatexal 7y ago
And on architectures like EPYC you run into NUMA issues and having to pin things to certain cores, right?
- avar 7y ago
  Yes, I wanted to keep it short, but you can get pathological performance on such systems unless you carefully use the likes of taskset(1) or numactl(1) to smartly pin certain classes of jobs to a given CPU. This is a decent reference: https://www.glennklockwood.com/hpc-howtos/process-affinity.html https://www.glennklockwood.com/hpc-howtos/process-affinity.h...
- theevilsharpie 7y ago
  The OS will attempt to schedule tasks as close to their memory as possible. Pinning tasks to specific cores may be needed in certain workloads, but for loosely-coupled parallel tasks like compiling code, you'll do fine letting the OS do its thing.
- vbezhenar 7y ago
  make launches separate processes, so NUMA should not be an issue, there's no interprocess communication.
  - avar 7y ago
    Generally NUMA doesn't matter for "make" workloads so this as all academic, but that being said no, this isn't how modern CPUs work. When you start two unrelated processes one after the other (such as multiple compile steps) that operate on some of the same in-memory assets (files, things being sent over a pipe etc.) they're not just going to be in the main RAM, but also L1-3 cache, and the RAM itself may be segmented under the hood (even if it's presented to you as one logical address space). Thus you can benefit from pinning certain groups of tasks to a given CPU/memory space if you know the caches can be re-used without the OS having to transfer the memory to another CPU's purview, or re-populate the relevant caches from RAM.

by avar 7y ago

gigatexal 7y ago

And on architectures like EPYC you run into NUMA issues and having to pin things to certain cores, right?

avar 7y ago
Yes, I wanted to keep it short, but you can get pathological performance on such systems unless you carefully use the likes of taskset(1) or numactl(1) to smartly pin certain classes of jobs to a given CPU. This is a decent reference: https://www.glennklockwood.com/hpc-howtos/process-affinity.html https://www.glennklockwood.com/hpc-howtos/process-affinity.h...
theevilsharpie 7y ago
The OS will attempt to schedule tasks as close to their memory as possible. Pinning tasks to specific cores may be needed in certain workloads, but for loosely-coupled parallel tasks like compiling code, you'll do fine letting the OS do its thing.
vbezhenar 7y ago
make launches separate processes, so NUMA should not be an issue, there's no interprocess communication.
- avar 7y ago
  Generally NUMA doesn't matter for "make" workloads so this as all academic, but that being said no, this isn't how modern CPUs work. When you start two unrelated processes one after the other (such as multiple compile steps) that operate on some of the same in-memory assets (files, things being sent over a pipe etc.) they're not just going to be in the main RAM, but also L1-3 cache, and the RAM itself may be segmented under the hood (even if it's presented to you as one logical address space). Thus you can benefit from pinning certain groups of tasks to a given CPU/memory space if you know the caches can be re-used without the OS having to transfer the memory to another CPU's purview, or re-populate the relevant caches from RAM.