4 ms·

I'd assume that 'warp' is taken from textiles: https://en.m.wikipedia.org/wiki/Warp_and_weft https://en.m.wikipedia.org/wiki/Warp_and_weft A warp is a thread,

I'd assume that 'warp' is taken from textiles: https://en.m.wikipedia.org/wiki/Warp_and_weft https://en.m.wikipedia.org/wiki/Warp_and_weft A warp is a thread, but a thread within a matrix of other threads. I'm not into GPU programming, but doesn't nvidia have some notion of arranging threads in a matrix sort of like this?

adrian_b 2y ago
Nope. In NVIDIA parlance, a thread is the body of a "parallel for" structure, i.e. the sequence of operations that are executed for an array element, which are executed by one SIMD lane of a GPU. A "warp" is a set of "threads", normally of 32 "threads" for the NVIDIA GPUs, the number of "threads" in a "warp" being the number of SIMD lanes of the execution units. CUDA uses what Hoare (1978) has named "array of processes" and which in many programming languages is named "parallel for" or "parallel do". This looks like a "for" loop, but its body is not executed sequentially in a loop, but the execution is performed concurrently for all elements of the array. A modern CPU or GPU consists of many cores, each cores can execute multiple threads and each thread can execute SIMD instructions that perform an operation for multiple array elements, on distinct SIMD lanes. When a parallel for is launched in execution, the array elements are distributed over all existing cores, threads and SIMD lanes. In the case of NVIDIA, the distribution is handled by the CUDA driver, so it is transparent for the programmer, who does not have to know the structure of the GPU. The use by NVIDIA of the word "thread" would have corresponded with the reality if the GPU would not have used SIMD execution units. Real GPUs use SIMD instructions that process a number of array elements typically between 16 and 64. NVIDIA's "warp" is the real thread executed by the GPU, which processes multiple array elements, while NVIDIA's "thread" is what would have been executed by a thread of a fictitious GPU that does not use SIMD, so it would process only one array element per thread.
- mystified5016 2y ago
  I dunno, it still sounds to me that nvidia is taking their (admittedly inaccurate) concept of a thread, putting a bunch in parallel, and calling that a warp to be cute. I think the analogy still makes a kind of sense if you accept it at face value and not worry about the exact definitions. Which is really all it needs to do, IMO. Again, I don't really know anything about GPUs, just speculating on the analogy.
  - gregw2 2y ago
    Agreed that warp is a marketing term, but it is definitely not something that should be called "threads" except in the very loosest sense of the term. A bunch of threads in parallel implies MIMD parallelism- multiple instructions multiple data. A warp implies SIMD parallelism - single instruction multiple data (although technically SIMT, single instructions multiple threads https://en.wikipedia.org/wiki/Single_instruction,_multiple_threads https://en.wikipedia.org/wiki/Single_instruction,_multiple_t...). From both a hardware and software perspective those are very different types of parallelism that Nvidia's architects and the architects of its predecessors at Sun/SGI/Cray/elsewhere were intimately familiar with. See: https://en.wikipedia.org/wiki/Flynn%27s_taxonomy https://en.wikipedia.org/wiki/Flynn%27s_taxonomy

adrian_b 2y ago

Nope. In NVIDIA parlance, a thread is the body of a "parallel for" structure, i.e. the sequence of operations that are executed for an array element, which are executed by one SIMD lane of a GPU. A "warp" is a set of "threads", normally of 32 "threads" for the NVIDIA GPUs, the number of "threads" in a "warp" being the number of SIMD lanes of the execution units. CUDA uses what Hoare (1978) has named "array of processes" and which in many programming languages is named "parallel for" or "parallel do". This looks like a "for" loop, but its body is not executed sequentially in a loop, but the execution is performed concurrently for all elements of the array. A modern CPU or GPU consists of many cores, each cores can execute multiple threads and each thread can execute SIMD instructions that perform an operation for multiple array elements, on distinct SIMD lanes. When a parallel for is launched in execution, the array elements are distributed over all existing cores, threads and SIMD lanes. In the case of NVIDIA, the distribution is handled by the CUDA driver, so it is transparent for the programmer, who does not have to know the structure of the GPU. The use by NVIDIA of the word "thread" would have corresponded with the reality if the GPU would not have used SIMD execution units. Real GPUs use SIMD instructions that process a number of array elements typically between 16 and 64. NVIDIA's "warp" is the real thread executed by the GPU, which processes multiple array elements, while NVIDIA's "thread" is what would have been executed by a thread of a fictitious GPU that does not use SIMD, so it would process only one array element per thread.

mystified5016 2y ago
I dunno, it still sounds to me that nvidia is taking their (admittedly inaccurate) concept of a thread, putting a bunch in parallel, and calling that a warp to be cute. I think the analogy still makes a kind of sense if you accept it at face value and not worry about the exact definitions. Which is really all it needs to do, IMO. Again, I don't really know anything about GPUs, just speculating on the analogy.
- gregw2 2y ago
  Agreed that warp is a marketing term, but it is definitely not something that should be called "threads" except in the very loosest sense of the term. A bunch of threads in parallel implies MIMD parallelism- multiple instructions multiple data. A warp implies SIMD parallelism - single instruction multiple data (although technically SIMT, single instructions multiple threads https://en.wikipedia.org/wiki/Single_instruction,_multiple_threads https://en.wikipedia.org/wiki/Single_instruction,_multiple_t...). From both a hardware and software perspective those are very different types of parallelism that Nvidia's architects and the architects of its predecessors at Sun/SGI/Cray/elsewhere were intimately familiar with. See: https://en.wikipedia.org/wiki/Flynn%27s_taxonomy https://en.wikipedia.org/wiki/Flynn%27s_taxonomy