3 ms·
One of the hard things about writing parallelized programs is that you may want different algorithms for different environments. The other day I joked about CUD
by godelski 3y ago
One of the hard things about writing parallelized programs is that you may want different algorithms for different environments. The other day I joked about CUDA being harder than rocket science[0] and this is part of it.
The thing with high parallelism is that you can't just think about your program in terms of clock and total memory. You have to consider all parts of the system, and this is likely the large reason most programs don't have high parallelism despite computers being many cores for decades. OpenMP is going to have different settings than OpenMPI. How you share the memory is an essential part of the algorithm you're going to write.
There's also issues about real cores and hyperthreads/logical. I find that in most computation I never want to use the number of logical cores but only rely on physical cores. You can also get a double descent like behavior[1]. The author here seems to be running into an even newer problem that is the difference between performance cores and efficiency cores. I'm not actually sure that this is surprising and makes the headline feel clickbaity.
For more, this is actually part of why CUDA is making a big boom lately. One of my first forays into CUDA was trying to write some kernels to speed up GEANT4 simulations, which are not too dissimilar from path tracing but they are more computationally heavy. So extremely parallelization but the operations per cycle wasn't high enough back then so it was still better to use CPU (I'm sure there were other bottlenecks as well but did find someone else confirming my results). This goes for lots of scientific compute too, like a lot of mesh solvers and FEA (finite element analysis: how you simulate physical parts). But not IPC has gone way up (AND cores increase!! AND bandwith! AND memory!) so now we start seeing these things start to leverage graphics cards a lot more.
In HPC and it is worth noting that the big bottleneck actually isn't compute, it's throughput. More data can be created than can be processed. Many teams are switching to "in situ" visualization/analysis methods where you offload your data to another machine which does that processing. You also simulate at FAR higher resolution than you visualize or even analyze. So if you're interested in tackling problems that are becoming more and more important, this is one of them and involves both hardware and software.
[0] https://news.ycombinator.com/item?id=38678066 https://news.ycombinator.com/item?id=38678066
[1] Say you have 16 physical cores with hyperthreading so 32 logical cores. Your program speeds up as number of processes -> 16. But when you go to 17 cores you have a big jump (loss in speedup) and follow a shallower curve as number of processes -> 32.
Edit: If you're not on an HPC machine, I actually might suggest setting your max parallelism to (number_of_physical_cores - 1) because speed difference often is trivial (lots of asterisks here) and you have a core available to kill the process. More noob you are, the stronger this recommendation.