5 ms·

make -j24

vetinari 7y ago
Be sure to have enough RAM, so you will not run out of it. (For some reason, when building projects like LLVM with -j 16, 32 GB w/o swap may be not enough. With -j 2, it is enough, but takes an eternity).
- nikic 7y ago
  If you build LLVM often, use the shared library build (BUILD_SHARED_LIBS=true). Most of the memory usage (and a large part of the time in incremental builds) comes from linking final artifacts if you do not use shared libraries.
  - vetinari 7y ago
    Thanks for the tip, unfortunately, when I'm building it (occasionally), the LLVM itself is part of another project (AMDVLK) and changes too.
- gigatexal 7y ago
  The initial plan is for a 32GB (2x 16GB sticks) with plans to move to 64GB when possible.
- 0815test 7y ago
  Indeed, if you're bottlenecked on RAM, memory bandwidth itself will also be an issue (it seems to be the foremost bottleneck in modern compute, outside of single-threaded workloads. Part of why C-like languages are again becoming popular these days - they economize on memory-bandwidth per core). Might want to skip Ryzen 9 altogether and wait until the Threadripper parts are announced.
- tlamponi 7y ago
  Yes, building Ceph with less than 24 GB ram on a 16 core machine will make it run into Out Of Memory situations here, so that the OOM-Killer is summoned and kills the build..
- jjuhl 7y ago
  Make sure to install and use ccache and give it a big cache size on a fast ssd (I use 96GiB). It speeds up rebuilds (after the cache is populated) quite significantly.
avar 7y ago
Having 24 threads doesn't mean that -j24 is the sweet spot. I have a machine 56 thread Xenon machine for building git.git, and I find that with its sockets/threads-per-core/threads-per-socket of 2/2/14 the sweet spot is closer to sockets*threads-per-socket = -j28. Things speed up rather linearly up to -j28, but once I get past -j28 (say -j32) it levels off, and -j56 starts being counterproductive. Same thing with the 160 thread POWER8 machine I have access to. That one runs 8 threads per core, and CPU-limited -jN tops out at around -j20. All of this is very worflow and CPU specific, but generally speaking don't blindly trust what things like "htop" show you as the number of available CPUs, under the hood many of them aren't "real".
- gigatexal 7y ago
  And on architectures like EPYC you run into NUMA issues and having to pin things to certain cores, right?
  - avar 7y ago
    Yes, I wanted to keep it short, but you can get pathological performance on such systems unless you carefully use the likes of taskset(1) or numactl(1) to smartly pin certain classes of jobs to a given CPU. This is a decent reference: https://www.glennklockwood.com/hpc-howtos/process-affinity.html https://www.glennklockwood.com/hpc-howtos/process-affinity.h...
  - theevilsharpie 7y ago
    The OS will attempt to schedule tasks as close to their memory as possible. Pinning tasks to specific cores may be needed in certain workloads, but for loosely-coupled parallel tasks like compiling code, you'll do fine letting the OS do its thing.
  - vbezhenar 7y ago
    make launches separate processes, so NUMA should not be an issue, there's no interprocess communication.
    - 7y ago
chrisseaton 7y ago
Do you have the memory and IO bandwidth to back that up?

make -j24

Be sure to have enough RAM, so you will not run out of it. (For some reason, when building projects like LLVM with -j 16, 32 GB w/o swap may be not enough. With -j 2, it is enough, but takes an eternity).

nikic 7y ago
If you build LLVM often, use the shared library build (BUILD_SHARED_LIBS=true). Most of the memory usage (and a large part of the time in incremental builds) comes from linking final artifacts if you do not use shared libraries.
- vetinari 7y ago
  Thanks for the tip, unfortunately, when I'm building it (occasionally), the LLVM itself is part of another project (AMDVLK) and changes too.
gigatexal 7y ago
The initial plan is for a 32GB (2x 16GB sticks) with plans to move to 64GB when possible.
0815test 7y ago
Indeed, if you're bottlenecked on RAM, memory bandwidth itself will also be an issue (it seems to be the foremost bottleneck in modern compute, outside of single-threaded workloads. Part of why C-like languages are again becoming popular these days - they economize on memory-bandwidth per core). Might want to skip Ryzen 9 altogether and wait until the Threadripper parts are announced.
tlamponi 7y ago
Yes, building Ceph with less than 24 GB ram on a 16 core machine will make it run into Out Of Memory situations here, so that the OOM-Killer is summoned and kills the build..
jjuhl 7y ago
Make sure to install and use ccache and give it a big cache size on a fast ssd (I use 96GiB). It speeds up rebuilds (after the cache is populated) quite significantly.

avar 7y ago

Having 24 threads doesn't mean that -j24 is the sweet spot. I have a machine 56 thread Xenon machine for building git.git, and I find that with its sockets/threads-per-core/threads-per-socket of 2/2/14 the sweet spot is closer to sockets*threads-per-socket = -j28. Things speed up rather linearly up to -j28, but once I get past -j28 (say -j32) it levels off, and -j56 starts being counterproductive. Same thing with the 160 thread POWER8 machine I have access to. That one runs 8 threads per core, and CPU-limited -jN tops out at around -j20. All of this is very worflow and CPU specific, but generally speaking don't blindly trust what things like "htop" show you as the number of available CPUs, under the hood many of them aren't "real".

gigatexal 7y ago
And on architectures like EPYC you run into NUMA issues and having to pin things to certain cores, right?
- avar 7y ago
  Yes, I wanted to keep it short, but you can get pathological performance on such systems unless you carefully use the likes of taskset(1) or numactl(1) to smartly pin certain classes of jobs to a given CPU. This is a decent reference: https://www.glennklockwood.com/hpc-howtos/process-affinity.html https://www.glennklockwood.com/hpc-howtos/process-affinity.h...
- theevilsharpie 7y ago
  The OS will attempt to schedule tasks as close to their memory as possible. Pinning tasks to specific cores may be needed in certain workloads, but for loosely-coupled parallel tasks like compiling code, you'll do fine letting the OS do its thing.
- vbezhenar 7y ago
  make launches separate processes, so NUMA should not be an issue, there's no interprocess communication.
  - 7y ago

chrisseaton 7y ago

Do you have the memory and IO bandwidth to back that up?