3 ms·
Your comment is very high quality, despite being short. > Extracting enough fine grain parallelism is hard. Manually creating it is tedious and difficult. A b
by dragontamer 5y ago
Your comment is very high quality, despite being short.
> Extracting enough fine grain parallelism is hard. Manually creating it is tedious and difficult.
A big win for NVidia was tricking enough programmers into manually describing large swaths of parallelism available for a GPU to take advantage of.
Manually creating fine grained parallelism through classical structures (mutex, semaphores, async, etc. etc.) is tedious and probably counterproductive. But it seems like describing __some__ extremely common forms of parallelism is in fact very useful in certain applications (matrix multiplications).
OpenMP, CUDA, OpenCL, (and even Fortran/Matlab to some extent)... even Tensorflow... are showing easy ways to describe a parallel problem.
CPUs seem to be the best architecture at discovering latent parallelism in general purpose code that the original programmer wasn't aware of. (Typical programmers are not aware of the multiple-pipelines that a modern CPU tries to fill up... and yet that code continues to accelerate from generation-to-generation... now with Apple's M1 proving that even 8-way decode is reasonable and provides performance gains). CPU designers seem to be experts at finding fine-grain parallelism as it is, and I don't think anyone can beat them at it. (Only another CPU designer will probably make something better).
---------
But for these alternative compute paradigms: such as GPU-compute, there are far fewer programmers and compiler experts working on that architecture. Its far easier to find little gold-nuggets here and there that lead to improved execution.
SIMD-languages such as C-Star, Lisp-Star, CUDA, OpenCL, OpenMP... have been describing parallel patterns for decades now. And yet, it only feels like we've begun to scratch that surface.