3 ms·
> Building software Well, the problem with that is that no make system can utilise so much core. Even linux kernel, probably the biggest C project in mainstrea
by baybal2 6y ago
> Building software
Well, the problem with that is that no make system can utilise so much core. Even linux kernel, probably the biggest C project in mainstream use, can not consistently load even 16 cores with mostly handwritten makefiles.
P.S.
I do not say that building software is not CPU parallelisable in principles, I'm saying that regular make systems have trouble handing big number of parallel tasks in practice, even in big, well run projects that use handwritten makefiles.
- ajross 6y agoThat's... not true at all. A kernel build will be as parallelizable as you like. I do kernel builds fairly regularly on 56 and 64 core hosts and in fact the build time is quite linear with core count. (The ramdisk packing provided by the distro at "make install" time tends to be what hurts scaling, FWIW). And my day job involves lots of iterations over the Zephyr test suite, which builds and runs hundreds of individual test apps X dozens of platforms. This is almost a pure build-time problem and scales very linearly indeed.
- baybal2 6y agoNot in my experience, from time to time it does boost to 40+ compile tasks, especially on drivers code, but it mostly hover at around 10 tasks.
- CrazyStat 6y agoCPU-parallelizable and GPU-parallelizable are different problems. CPU cores are independent. GPU cores are SIMD--every core does the same operation, but with different data. That works really well for things like matrix multiplication where you need to multiply a million times and then add a million times, because you can have all 3000 CUDA cores multiplying and then all 3000 CUDA cores adding. It doesn't work for compiling software.
- monocasa 6y agoIt's not that SIMD for GPUs. It'll be 64 or 32 GPU threads together generally sharing the dispatch and control logic. So if you have a 4096 "CUDA core" GPU, it'll be 64 of what would be called cores on the CPU side, each with a 64 lane vector processor.
- emidln 6y agoBazel utilizes the cores near linearly. It even makes use of remote cores in an efficient manner for compiling c++. 128 thread and 256 thread machines are great if you also have enough ddr4 to let each gcc/clang do their thing.
- pdimitar 6y agoThe rumoured Threadripper Pro 3995WC (64c/128t but with 8-channel memory support) might solve that problem.
- jchw 6y agoSomething is wrong on your end, you can parallelize the build of typical C and C++ software almost infinitely. The reason why is because most C and C++ software do not have 'happens-before' dependency relationships due to the way translation units work, so basically every translation unit in a module can be compiled 100% independently of any other translation unit. In practice that means you can easily scale up to 1000+ processes with no issue, and in fact, this has been accomplished in combination with AWS Lambda: https://stanford.edu/~sadjad/gg-paper.pdf https://stanford.edu/~sadjad/gg-paper.pdf Actual screencast of it in action compiling FFMPEG: https://asciinema.org/a/257545 https://asciinema.org/a/257545 (Perhaps the issue is you are bottlenecking on I/O or some other resource?)