I like to put it like this: a GPU is a really, really fast Map-Reduce hardware processor. You give it a bunch of data with elements which are all independent o

I like to put it like this: a GPU is a really, really fast Map-Reduce hardware processor. You give it a bunch of data with elements which are all independent of each other, apply the same transformations to to each element, and output a single data structure (a colour attachment or render target, or some compute buffer). This explains a lot of the shortcomings and design decisions of GPUs. They prioritise throughput over latency; they have a lot of memory but much slower than the cores themselves (~500 – 700 cycle latency compared to ~20 – 40 cycle latency on the CPU); they run 'threads' in synchronised lock-step, called warps; branching on a GPU is particularly expensive—if one shader for one fragment needs to branch, then the entire warp that it's on will have to wait for it, or worse still, calculate that branch, too; Their 3D graphics legacy still endures: despite the advent of GPGPU, they still have plenty of dedicated hardware for the graphics pipeline, including rasterisers, texture-mapping units, render output processors, etc. Once people get this idea, then GPU programming is easier to reason about. One execution of a shader deals with one fragment (actually four, but at a high level of abstraction, this is sufficient). Oddly enough, I did GPU parallelisation first, and then came to CPU parallelisation. I find CPU-level parallelism and concurrency problems like OS inter-process synchronisation, async-await, etc. much harder to reason about, because they generally cannot be reduced to such a simplistic 'one thread for one pixel'. It's also why I like embarrassingly parallel problems, because they're the most easily GPU-able.