> Most programmers have an intimate understanding of CPUs Not at all. Very few programmers have an intimate understanding of CPUs. I'm not one of them, for exa

> Most programmers have an intimate understanding of CPUs Not at all. Very few programmers have an intimate understanding of CPUs. I'm not one of them, for example. And that's without mentioning how most programmers actually write Excel scripts, and most of the rest write interpreted code like Javascript or Python which is somewhat removed from the CPU. > because they grow up writing code for the CPU There is not really such thing as "The CPU". The design space of CPUs is large enough for it to contain quite a variation of beasts. For example, > CPUs were designed to execute sequential instructions Originally, yes, but - one cannot really make this argument about CPUs since Dennard scaling hit the wall, and scale-out began in earnest. ----- While the mental model is indeed different, hardware-wise, you should think of GPUs as CPUs, except that... * You only ever use SIMD registers: 32 lanes x 32bit * Large register file (up to 256 SIMD registers) * No branch prediction * Predicated execution supported, but no if-then-else (sort of). * Very little stack to speak of * ~10x L1 cache latency * A piece of L1 "cache" which doesn't cache anything and you can just write to it like regular memory. * ~100x L2 cache latency, no coherence, and it's smaller * ~100x main memory latency (and no L3) * A bunch of special registers for implementing geometric thread & lane indexing. * ~10x memory bandwidth * ~5x-10x cores than on a CPU (numbers are for NVIDIA cards, AMD are slightly different; and of course individual cards differ etc.) ----- Now take all of those hardware differences, and conceive a programming model. What you get is the fanciful tale of a "not-like-a-CPU" processor. Programming for the CPU, we often do the opposite, i.e. ignore the cores we have, ignore things like latency hiding, careful management of what's in registers and what's in the cache, and just write naive serial programs.