6 ms·
What will you be doing that will actually use 24 threads?
by WC3w6pXxgGd 7y ago
What will you be doing that will actually use 24 threads?
- ubercow13 7y agomake -j24
- vetinari 7y agoBe sure to have enough RAM, so you will not run out of it. (For some reason, when building projects like LLVM with -j 16, 32 GB w/o swap may be not enough. With -j 2, it is enough, but takes an eternity).
- nikic 7y agoIf you build LLVM often, use the shared library build (BUILD_SHARED_LIBS=true). Most of the memory usage (and a large part of the time in incremental builds) comes from linking final artifacts if you do not use shared libraries.
- vetinari 7y agoThanks for the tip, unfortunately, when I'm building it (occasionally), the LLVM itself is part of another project (AMDVLK) and changes too.
- gigatexal 7y agoThe initial plan is for a 32GB (2x 16GB sticks) with plans to move to 64GB when possible.
- 0815test 7y agoIndeed, if you're bottlenecked on RAM, memory bandwidth itself will also be an issue (it seems to be the foremost bottleneck in modern compute, outside of single-threaded workloads. Part of why C-like languages are again becoming popular these days - they economize on memory-bandwidth per core). Might want to skip Ryzen 9 altogether and wait until the Threadripper parts are announced.
- tlamponi 7y agoYes, building Ceph with less than 24 GB ram on a 16 core machine will make it run into Out Of Memory situations here, so that the OOM-Killer is summoned and kills the build..
- jjuhl 7y agoMake sure to install and use ccache and give it a big cache size on a fast ssd (I use 96GiB). It speeds up rebuilds (after the cache is populated) quite significantly.
- avar 7y agoHaving 24 threads doesn't mean that -j24 is the sweet spot. I have a machine 56 thread Xenon machine for building git.git, and I find that with its sockets/threads-per-core/threads-per-socket of 2/2/14 the sweet spot is closer to sockets*threads-per-socket = -j28. Things speed up rather linearly up to -j28, but once I get past -j28 (say -j32) it levels off, and -j56 starts being counterproductive. Same thing with the 160 thread POWER8 machine I have access to. That one runs 8 threads per core, and CPU-limited -jN tops out at around -j20. All of this is very worflow and CPU specific, but generally speaking don't blindly trust what things like "htop" show you as the number of available CPUs, under the hood many of them aren't "real".
- gigatexal 7y agoAnd on architectures like EPYC you run into NUMA issues and having to pin things to certain cores, right?
- avar 7y agoYes, I wanted to keep it short, but you can get pathological performance on such systems unless you carefully use the likes of taskset(1) or numactl(1) to smartly pin certain classes of jobs to a given CPU. This is a decent reference: https://www.glennklockwood.com/hpc-howtos/process-affinity.html https://www.glennklockwood.com/hpc-howtos/process-affinity.h...
- theevilsharpie 7y agoThe OS will attempt to schedule tasks as close to their memory as possible. Pinning tasks to specific cores may be needed in certain workloads, but for loosely-coupled parallel tasks like compiling code, you'll do fine letting the OS do its thing.
- vbezhenar 7y agomake launches separate processes, so NUMA should not be an issue, there's no interprocess communication.
- 7y ago
- chrisseaton 7y agoDo you have the memory and IO bandwidth to back that up?
- fnord123 7y agoRun electron apps.
- lagadu 7y agoImplying that there's a computer in existence that runs those fast.
- sundvor 7y agoAwesome, I just realised that the old "run Crysis" joke has been replaced. Electron apps do manage to make a mockery of my PC's specs though.
- gigatexal 7y agoI know this is a joke but aren't Electron apps single threaded by nature and usually memory hogs? So more memory would be better and having many cores is good too.
- 781 7y agoParent was asked about threads, not memory. Of course he also has 1 TB to keep those Electron apps happy.
- gigatexal 7y agohaha, burn!
- fnord123 7y agoElectron apps have Helper processes. e.g. I currently have 3 Microsoft Teams Helper processes (with 37, 36, and 16 threads) running and 3 Spotify Helper processes (with 16, 9, and 4 threads).
- megous 7y agoIdle threads don't count.
- bayindirh 7y agoDevelop multi-threaded scientific applications?
- gameswithgo 7y agoA common developer scenario can involve a few things that will eat a lot of threads: 1. playing some background music 2. running a local database 3. running a local webserver 4. running a browser 5. running an ide 6. running all of that stuff concurrently while testing the backend code 7. doing builds which are multi threaded in near linear speedup fashion in many languages/environments. I don't know if 12 cores 24 logical is going to make that scenario feel overall better than 4 cores 8 logical, but I do know that 4x8 feels much much better than 2x4 in my own use cases. #7 alone can be a really, really big win for long compiling projects.
- gigatexal 7y agoThis. But to the OP's main criticism: I could very well do all of that with 16-threads and 8 real cores. I do a lot of work currently with distributed databases and I believe I need the cores for local testing along with everything else that gameswithgo mentioned.
- dkersten 7y agoYeah, my docker setup alone runs a ton of processes. Aside from docker running a copy of my stuff, I often have tests auto-running, a separate REPL to try stuff out in, my editor, slack, music, browser. It all adds up and a bunch of cores/threads definitely makes everything run more smoothly.
- Traster 7y agoCommon developer scenario: make -j24
- api 7y agoI regularly max out 8 as a developer and could certainly make use of 16 or 24, and I am probably toward the moderate end of developer needs. Examples: multiple VMs, big editors/IDEs, local databases, local k8s clusters, local network simulators, and dont even start with AI or big analytics stuff.