5 ms·

Not even SIMD on recent GPUs, it's SIMT.

Jasper_ 6y ago
SIMT has always just a fancy disguise on top of SIMD hardware, but don't tell the GPU manufacturers I said that :)
dragontamer 6y ago
No one has ever been able to tell me the difference between SIMT and 1980s style SIMD on the Connection Machine 5 (CM-5). Researchers like Blelloch have been writing in SIMD-parallel style "parallel Lisp" or "C-Star" for decades before CUDA. https://www.cs.cmu.edu/~guyb/papers/Ble90.pdf https://www.cs.cmu.edu/~guyb/papers/Ble90.pdf
- dahart 6y ago
  Thread masking is one of the big differences - threads in a warp today can be enabled or disabled independently to enable limited amounts of divergent flow control, and conversely super cheap syncronization. I don't know what the CM5 did - was it limited to always-on sets of vector instructions? Current GPUs also support certain amounts of fully independent thread scheduling, e.g., per-thread instruction counters, etc. Maybe that is beginning to leak out of the SIMT model.
  - dragontamer 6y ago
    > I don't know what the CM5 did - was it limited to always-on sets of vector instructions? EDIT: Woops, I mean CM2. CM5 was MIMD and a slightly different architecture. I've edited this post to say CM-2 instead, but my previous post about CM5 will remain in error. Individual thread masking was available on CM-2, for all 4096 cores that were executing in SIMD-parallel. CM2 was a 4096 x 1-bit SIMD processor. Very limited compared to modern GPUs, but the execution model that CM2 experimented with eventually became the modern GPU architecture. Yes, you can have effective execution masks even with only 1-bit cores. EDIT: See the C-Star manuals, a "per-thread if statement" required a "where" construct instead of if. But yes, the "where" statement performed similarly to "CUDA if": http://people.csail.mit.edu/bradley/cm5docs/AReferenceDescriptionoftheCStarLanguage.pdf http://people.csail.mit.edu/bradley/cm5docs/AReferenceDescri... The MIMD thing that CM-5 did didn't seem to stick. CM-2's SIMD execution seems to be a bigger influence. But C-Star ran on both... and it was the high-level languages like C-Star or Parallel-Lisp that influenced GPU languages like CUDA. > The where statement is involved with setting the context, a process known as contextualization. The context is a parallel boolean mask (i.e., each element of it is true or false) that controls the execution of parallel operations position by position. A different context is associated with each shape object, and the context associated with the current shape is always applied to operators. A CUDA-block was roughly equivalent to a CStar-Shape. A CUDA-if is roughly equivalent to a CStar-where. > Current GPUs also support certain amounts of fully independent thread scheduling, e.g., per-thread instruction counters, etc. Maybe that is beginning to leak out of the SIMT model. NVidia was calling SIMT back in 2010, long before per-thread instruction counters were available in Pascal (GTX 10xx series of GPUs)
    - dahart 6y ago
      Thanks for the C-star link. At first glance shapes look similar to what have become launch dimensions. I think it'd be pretty interesting to hear a comparison from someone who's used both C-star and CUDA, about the plusses and minuses of each... have you used both? > Very limited compared to modern GPUs Yeah, I'd expect the real true differences between CM5 SIMD and Pascal SIMT is just buried in the subtleties of those limitations compared to today's hardware, and that fundamentally they aren't super different ideas. In my mind, that doesn't mean that SIMT is meaningless or the same thing as SIMD. > NVidia was calling SIMT back in 2010, long before per-thread instruction counters were available in Pascal (GTX 10xx series of GPUs) Right; I'm saying the "SIMT" of 10 years ago has to do with masking and maybe other per-thread control (even though the Connection Machines might have had it as well), and that per-thread counters are now going beyond what we used to know as SIMT, perhaps deserving of some other acronym. Or maybe SIMT is becoming more appropriate and more differentiated from SIMD over time? Is it possible that CM5 should be called SIMT and we simply didn't have that acronym at the time?
      - dragontamer 6y ago
        > Thanks for the C-star link. At first glance shapes look similar to what have become launch dimensions. I think it'd be pretty interesting to hear a comparison from someone who's used both C-star and CUDA, about the plusses and minuses of each... have you used both? Nope. I'm looking at C-Star through a historical lens, I was curious at the overall development of CUDA and was researching the "line of influence", so to speak. > Right; I'm saying the "SIMT" of 10 years ago has to do with masking and maybe other per-thread control (even though the Connection Machines might have had it as well), and that per-thread counters are now going beyond what we used to know as SIMT, perhaps deserving of some other acronym. Or maybe SIMT is becoming more appropriate and more differentiated from SIMD over time? Is it possible that CM5 should be called SIMT and we simply didn't have that acronym at the time? CM2 was just another SIMD machine, there were plenty of others before CM2 (its just that CM2 was one of the more popular ones). IIRC, Cray had similar SIMD machines that competed against them. The overall SIMD-methodology reaches back until the 70s at least, maybe earlier. Just noting that CM5 was MIMD at the lowest level, trying to correct a mistake I made a few posts back. ------------ I think what "happened" was that Intel adopted SWAR: "SIMD With A Register" when Intel created the MMX / SSE / AVX instruction sets. (compared to dedicated SIMD-computers like the CM-2). At some point, Intel's SWAR approach became colloquially known as SIMD (even though SWAR was much more limited compared to "proper" SIMD computers like the CM2). NVidia creates a new acronym called SIMT to differentiate Intel's SIMD (aka: SWAR) from a proper SIMD machine, even though the 80s-style SIMD was substantially similar to NVidia's new SIMT term.

SIMT has always just a fancy disguise on top of SIMD hardware, but don't tell the GPU manufacturers I said that :)

No one has ever been able to tell me the difference between SIMT and 1980s style SIMD on the Connection Machine 5 (CM-5). Researchers like Blelloch have been writing in SIMD-parallel style "parallel Lisp" or "C-Star" for decades before CUDA. https://www.cs.cmu.edu/~guyb/papers/Ble90.pdf https://www.cs.cmu.edu/~guyb/papers/Ble90.pdf

dahart 6y ago
Thread masking is one of the big differences - threads in a warp today can be enabled or disabled independently to enable limited amounts of divergent flow control, and conversely super cheap syncronization. I don't know what the CM5 did - was it limited to always-on sets of vector instructions? Current GPUs also support certain amounts of fully independent thread scheduling, e.g., per-thread instruction counters, etc. Maybe that is beginning to leak out of the SIMT model.
- dragontamer 6y ago
  > I don't know what the CM5 did - was it limited to always-on sets of vector instructions? EDIT: Woops, I mean CM2. CM5 was MIMD and a slightly different architecture. I've edited this post to say CM-2 instead, but my previous post about CM5 will remain in error. Individual thread masking was available on CM-2, for all 4096 cores that were executing in SIMD-parallel. CM2 was a 4096 x 1-bit SIMD processor. Very limited compared to modern GPUs, but the execution model that CM2 experimented with eventually became the modern GPU architecture. Yes, you can have effective execution masks even with only 1-bit cores. EDIT: See the C-Star manuals, a "per-thread if statement" required a "where" construct instead of if. But yes, the "where" statement performed similarly to "CUDA if": http://people.csail.mit.edu/bradley/cm5docs/AReferenceDescriptionoftheCStarLanguage.pdf http://people.csail.mit.edu/bradley/cm5docs/AReferenceDescri... The MIMD thing that CM-5 did didn't seem to stick. CM-2's SIMD execution seems to be a bigger influence. But C-Star ran on both... and it was the high-level languages like C-Star or Parallel-Lisp that influenced GPU languages like CUDA. > The where statement is involved with setting the context, a process known as contextualization. The context is a parallel boolean mask (i.e., each element of it is true or false) that controls the execution of parallel operations position by position. A different context is associated with each shape object, and the context associated with the current shape is always applied to operators. A CUDA-block was roughly equivalent to a CStar-Shape. A CUDA-if is roughly equivalent to a CStar-where. > Current GPUs also support certain amounts of fully independent thread scheduling, e.g., per-thread instruction counters, etc. Maybe that is beginning to leak out of the SIMT model. NVidia was calling SIMT back in 2010, long before per-thread instruction counters were available in Pascal (GTX 10xx series of GPUs)
  - dahart 6y ago
    Thanks for the C-star link. At first glance shapes look similar to what have become launch dimensions. I think it'd be pretty interesting to hear a comparison from someone who's used both C-star and CUDA, about the plusses and minuses of each... have you used both? > Very limited compared to modern GPUs Yeah, I'd expect the real true differences between CM5 SIMD and Pascal SIMT is just buried in the subtleties of those limitations compared to today's hardware, and that fundamentally they aren't super different ideas. In my mind, that doesn't mean that SIMT is meaningless or the same thing as SIMD. > NVidia was calling SIMT back in 2010, long before per-thread instruction counters were available in Pascal (GTX 10xx series of GPUs) Right; I'm saying the "SIMT" of 10 years ago has to do with masking and maybe other per-thread control (even though the Connection Machines might have had it as well), and that per-thread counters are now going beyond what we used to know as SIMT, perhaps deserving of some other acronym. Or maybe SIMT is becoming more appropriate and more differentiated from SIMD over time? Is it possible that CM5 should be called SIMT and we simply didn't have that acronym at the time?
    - dragontamer 6y ago
      > Thanks for the C-star link. At first glance shapes look similar to what have become launch dimensions. I think it'd be pretty interesting to hear a comparison from someone who's used both C-star and CUDA, about the plusses and minuses of each... have you used both? Nope. I'm looking at C-Star through a historical lens, I was curious at the overall development of CUDA and was researching the "line of influence", so to speak. > Right; I'm saying the "SIMT" of 10 years ago has to do with masking and maybe other per-thread control (even though the Connection Machines might have had it as well), and that per-thread counters are now going beyond what we used to know as SIMT, perhaps deserving of some other acronym. Or maybe SIMT is becoming more appropriate and more differentiated from SIMD over time? Is it possible that CM5 should be called SIMT and we simply didn't have that acronym at the time? CM2 was just another SIMD machine, there were plenty of others before CM2 (its just that CM2 was one of the more popular ones). IIRC, Cray had similar SIMD machines that competed against them. The overall SIMD-methodology reaches back until the 70s at least, maybe earlier. Just noting that CM5 was MIMD at the lowest level, trying to correct a mistake I made a few posts back. ------------ I think what "happened" was that Intel adopted SWAR: "SIMD With A Register" when Intel created the MMX / SSE / AVX instruction sets. (compared to dedicated SIMD-computers like the CM-2). At some point, Intel's SWAR approach became colloquially known as SIMD (even though SWAR was much more limited compared to "proper" SIMD computers like the CM2). NVidia creates a new acronym called SIMT to differentiate Intel's SIMD (aka: SWAR) from a proper SIMD machine, even though the 80s-style SIMD was substantially similar to NVidia's new SIMT term.