While you can always bend a turing-complete language to do the logic, it doesn't mean your abstractions carry all the way down to hardware — there is "overhead"

While you can always bend a turing-complete language to do the logic, it doesn't mean your abstractions carry all the way down to hardware — there is "overhead" or "loss of efficiency" if you will. You get the convenience, and perhaps organize the safety; but it only goes so far. Well, hmm, I think you're right about the "hardware bending" part but unfortunately you are completely wrong about "Turing complete" part (I'm not sure where that comes in, tb). Indeed, a big factor with all this is that a c/c++ vector extension is pretty good reflection of a GPUs operations (and I'd assume a TPU, it being similar). To wit. If you look at the assembly code of a GPU (Nvidia PTX), appears is reasonably similar to the machine code of CPUs but some extra sorts of primitives to allow communication between kernels. Otherwise, the code is "SIMD" (or STMD as Nvidia says); Most of a program is a sequence of imperative instructions that are different from CPU instruction in that they're acting on a huge "vector" of values. You really don't "threads" in the fashion that CPUs have - as the GP says. What you talking about is a system that's absolutely "Turing complete" (or as Turing complete as assembler or C on a CPU) but with extra facilities. The parallelism involved is absolutely different from the parallelism of current CPU systems. This is why just everything is translated to c++ which is then translated to PTX. All that said, people have indeed proposed ways to bend GPUs to make them look more like current (MIMD) systems - see: https://www.microsoft.com/en-us/research/video/mimd-on-gpu/ https://www.microsoft.com/en-us/research/video/mimd-on-gpu/ etc. But if anything, the "bend the hardware argument" very much work against Haskell etc on the GPU because not all parallelism is concurrent system like.