4 ms·
This so-called GPU programming model has existed many decades before the appearance of the first GPUs, but at that time the compilers were not so good like the
by adrian_b 3y ago
This so-called GPU programming model has existed many decades before the appearance of the first GPUs, but at that time the compilers were not so good like the CUDA compilers, so the burden for a programmer was greater.
As another poster has already mentioned, there exists a compiler for CPUs which has been inspired by CUDA and which has been available for many years: ISPC (Implicit SPMD Program Compiler), at https://github.com/ispc/ispc https://github.com/ispc/ispc .
NVIDIA has the very annoying habit of using a lot of terms that are different from those that have been previously used in computer science for decades. The worst is that NVIDIA has not invented new words, but they have frequently reused words that have been widely used with other meanings.
SIMT (Single-Instruction Multiple Thread) is not the worst term coined by NVIDIA, but there was no need for yet another acronym. For instance they could have used SPMD (Single Program, Multiple Data Stream), which dates from 1988, two decades before CUDA.
Moreover, SIMT is the same thing that was called "array of processes" by C.A.R. Hoare in August 1978 (in "Communicating Sequential Processes"), or "replicated parallel" by Occam in 1985 or "PARALLEL DO" by "OpenMP Fortran" in 1997-10 or "parallel for" by "OpenMP C and C++" in 1998-10.
Each so-called CUDA kernel is just the body of a "parallel for" (which is multi-dimensional, like in Fortran).
The only (but extremely important) innovation brought by CUDA is that the compiler is smart enough so that the programmer does not need to know the structure of the processor, i.e. how many cores it has and how many SIMD lanes each core has. The CUDA compiler distributes automatically the work over the available SIMD lanes and available cores and in most cases the programmer does not care whether two executions of the function that must be executed for each data item are done on two different cores or on two different SIMD lanes of the same core.
This distribution of the work over SIMD lanes and over cores is simple when the SIMD operations are maskable, like in GPUs or in AVX-512 a.k.a. AVX10 or in ARM SVE. When masking is not available, like in AVX2 or Armv8-A, the implementation of conditional statements and expressions is more complicated.
- mathisfun123 3y ago[flagged]
- tovej 3y agoAre you even reading the text you cite? That compares SIMD to SIMT. SIMD, not SPMD. SIMD is a more restrictive hardware paradigm, SPMD is just a mutiple copies of a single program running on multiple processes/threads, i.e., exactly what CUDA does. Most MPI-programs are SPMD, and really, the only conceptual difference between MPI and CUDA is that a thread's id is a point in a grid space that's relevant for scheduling and memory locality. Oh wait, that's true for MPI as well, only the scheduling and memory locality relates to CPU cores and nodes, instead of CUDAs SMPs and blocks. The only real justification for a different programming model is that SIMT creates opportunities for implicit SIMD on a thread scheduling level if you've written your program in a way that the warps don't diverge.
- mathisfun123 3y ago> Are you even reading the text you cite? Yes I have read it enough times that I knew exactly where it was in the book. > That compares SIMD to SIMT. Yes and so does the op I responded to. > SIMD, not SPMD. SIMD is a more restrictive hardware paradigm, Thanks but I was already aware. The op was the one that conflated SPMD and SIMD, not me. > SPMD is just a mutiple copies of a single program running on multiple processes/threads, i.e., exactly what CUDA does. CUDA is a C language extension that enables you to write host and device code in the same source. That's it. SIMT is the model of compute on NVIDIA SMs. > The only real justification for a different programming model is that SIMT creates opportunities for implicit SIMD on a thread scheduling level if you've written your program in a way that the warps don't diverge. No that's incorrect. SIMT is implemented using predicated execution and instruction replay exactly so that warps can diverge. Again, it's amazing you guys are so sure that NVIDIA pulled a fast one on just the whole community/industry and didn't actually innovate but it turns out you're actually missing some pieces of the puzzle.
- tovej 3y agoOP did not conflate them, SPMD and SIMT are the same model, they're just used in different contexts.
- kllrnohj 3y agoDirectX 8 added programmable shaders as we mostly know it today in 2000 with directx 9's shader model 2.0 really solidifying things in 2002. CUDA didn't show up until 2007.
- pjmlp 3y agoExcept that shaders predate DirectX 8 by several decades. Two major examples, https://en.m.wikipedia.org/wiki/TMS34010 https://en.m.wikipedia.org/wiki/TMS34010 https://en.m.wikipedia.org/wiki/RenderMan_Shading_Language https://en.m.wikipedia.org/wiki/RenderMan_Shading_Language Also in DirectX 8, it wasn't as we know them today, because Assembly was the only programming language. Nowadays CUDA does C, C++, Fortran, Python JIT in the box, and has partner collaborations for Haskell, Java, Julia, C#.
- mattpharr 3y agoAlso related as far as the programming model: C* on Connection Machines and the C dialect used on MasPar systems in the 1980s. There is some discussion in the ispc paper: https://pharr.org/matt/assets/ispc.pdf https://pharr.org/matt/assets/ispc.pdf
- JonChesterfield 3y agoSIMT and SIMD are different things. It's fortunate that they have different names. A GPU is a single instruction multiple data machine. That's what the predicated vector operations are. 32 floats at a time, each with a disable bit. Cuda is a single instruction multiple thread language. You write code in terms of one float and branching on booleans, as if it was a CPU, with some awkward intrinsics for accessing the vector units in the GPU. That is, the programming model of a GPU ISA and that of Cuda are not the same. The GPU gives you vector instructions. Cuda gives you (mostly) scalar instructions and a compiler that deals with this mismatch, lowering branches to changes in exec mask and so forth. With my numerical library hat on, I hate this. Programming a simd machine through a simt language means trying to get the compiler to transform the control flow into the thing you could easily write using vector instructions. With my compiler implementer hat on, I hate this. It gives you two control flow graphs intertwined and a really bad time in register allocation. It's not totally clear to me why simt won out over writing the vector operations. I'm certainly in the minority opinion here.
- pca006132 3y agoI guess that a lot of people are uncomfortable thinking about vector instructions, and dealing with masks manually? And for vector instructions you need to align things properly, pad the arrays such that they are of the right size, that people are not used to I guess.
- RevEng 3y agoI agree. I work on numerical simulation software that involves very sparse, very irregular matrices. We hardly use SIMD because of the challenges of maintaining predicates, bitmasks, mapping values into registers, and so on. It's not bad if we can work on dense blocks, but dealing with sparsity is a huge headache. Now that we are implementing these methods with CUDA using the SIMT paradigm, that complexity is largely taken care of. We still need to consider how to design algorithms to have parallelism, but we don't have to manage all the minutiae of mapping things into and out of registers. Modern GPGPUs also have more hardware dedicated to this beyond the SIMD/SIMT models. In NVIDIAs CUDA programming model, besides the group of threads that represents a vector operation (a warp), you also have groups of warps (thread blocks) that are assigned the same processor and can explicitly address a fast, shared memory. Each processor has many registers that are automatically mapped to each thread so that each thread has its own dedicated registers. Scheduling is done in hardware at an instruction level so that you can effectively single cycle context switches between warps. Starting with Volta, it will even assemble vectors from threads in any warps in the same thread block, so lanes that are predicated off in a warp don't have to go to waste - they can take lanes from other warps. There are many other hardware additions that make this programming model very efficient. Similar to how C and x86 each provide abstractions over the actual micro ops being executed that hides complexity like pipelining, out of order execution, and speculative execution, CUDA and the PTX ISA provide abstractions over complex hardware implementations that specifically benefit this kind of SIMT paradigm.
- hasmanean 3y agoSubtle difference. A parallel_for can have asynchronous threads. They can all divergen and run independent instructions (due to if statements etc) SIMT means multiple processors all executing the exact same instruction. One instruction decoder, but like 64 execution pipelines.
- adrian_b 3y agoThe same is true for CUDA/OpenCL kernels. They can include conditionals and they can execute independent instructions. The difference is that because the software "kernels" (i.e. software threads) may be mapped by the compiler either on hardware threads or on hardware SIMD lanes, and this mapping is not controlled by the programmer, the divergent instructions will cause inefficient (serial) execution when they happen to be executed on SIMD lanes of the same core, so they must be avoided. This however is only an optimization problem, if the speed would be irrelevant all the kernels could execute divergent instructions. The reason for the existence of CUDA is to mask for the programmer the existence of the SIMD lanes and to allow programming as if the software threads would map only to hardware threads. Nevertheless, for optimum performance the programmer must be aware that this abstraction is not really true, so the programs should be written with awareness of the limitations introduced by SIMD.
- my123 3y ago> Nevertheless, for optimum performance On conventional SIMT implementations (pre-Volta), the programmer also has to be aware of it to not cause deadlocks in the atomics across different lanes in the same warp. On NV Volta onwards, each SIMT lane has its own instruction pointer with opportunistic reconvergence when possible.