4 ms·

One difficulty is that the "principles" aren't really pixels anymore, but things like triangles, rasterization, and SIMD programs that run. Understanding the hi

One difficulty is that the "principles" aren't really pixels anymore, but things like triangles, rasterization, and SIMD programs that run. Understanding the historical context of super old "graphics adapters" might not equip you for the (quite ingenious, imo) insights that underlie their transition into the "GPU".

darknoon 6y ago
Not even SIMD on recent GPUs, it's SIMT.
- Jasper_ 6y ago
  SIMT has always just a fancy disguise on top of SIMD hardware, but don't tell the GPU manufacturers I said that :)
- dragontamer 6y ago
  No one has ever been able to tell me the difference between SIMT and 1980s style SIMD on the Connection Machine 5 (CM-5). Researchers like Blelloch have been writing in SIMD-parallel style "parallel Lisp" or "C-Star" for decades before CUDA. https://www.cs.cmu.edu/~guyb/papers/Ble90.pdf https://www.cs.cmu.edu/~guyb/papers/Ble90.pdf
  - dahart 6y ago
    Thread masking is one of the big differences - threads in a warp today can be enabled or disabled independently to enable limited amounts of divergent flow control, and conversely super cheap syncronization. I don't know what the CM5 did - was it limited to always-on sets of vector instructions? Current GPUs also support certain amounts of fully independent thread scheduling, e.g., per-thread instruction counters, etc. Maybe that is beginning to leak out of the SIMT model.
    - dragontamer 6y ago
      > I don't know what the CM5 did - was it limited to always-on sets of vector instructions? EDIT: Woops, I mean CM2. CM5 was MIMD and a slightly different architecture. I've edited this post to say CM-2 instead, but my previous post about CM5 will remain in error. Individual thread masking was available on CM-2, for all 4096 cores that were executing in SIMD-parallel. CM2 was a 4096 x 1-bit SIMD processor. Very limited compared to modern GPUs, but the execution model that CM2 experimented with eventually became the modern GPU architecture. Yes, you can have effective execution masks even with only 1-bit cores. EDIT: See the C-Star manuals, a "per-thread if statement" required a "where" construct instead of if. But yes, the "where" statement performed similarly to "CUDA if": http://people.csail.mit.edu/bradley/cm5docs/AReferenceDescriptionoftheCStarLanguage.pdf http://people.csail.mit.edu/bradley/cm5docs/AReferenceDescri... The MIMD thing that CM-5 did didn't seem to stick. CM-2's SIMD execution seems to be a bigger influence. But C-Star ran on both... and it was the high-level languages like C-Star or Parallel-Lisp that influenced GPU languages like CUDA. > The where statement is involved with setting the context, a process known as contextualization. The context is a parallel boolean mask (i.e., each element of it is true or false) that controls the execution of parallel operations position by position. A different context is associated with each shape object, and the context associated with the current shape is always applied to operators. A CUDA-block was roughly equivalent to a CStar-Shape. A CUDA-if is roughly equivalent to a CStar-where. > Current GPUs also support certain amounts of fully independent thread scheduling, e.g., per-thread instruction counters, etc. Maybe that is beginning to leak out of the SIMT model. NVidia was calling SIMT back in 2010, long before per-thread instruction counters were available in Pascal (GTX 10xx series of GPUs)
      - dahart 6y ago
        Thanks for the C-star link. At first glance shapes look similar to what have become launch dimensions. I think it'd be pretty interesting to hear a comparison from someone who's used both C-star and CUDA, about the plusses and minuses of each... have you used both? > Very limited compared to modern GPUs Yeah, I'd expect the real true differences between CM5 SIMD and Pascal SIMT is just buried in the subtleties of those limitations compared to today's hardware, and that fundamentally they aren't super different ideas. In my mind, that doesn't mean that SIMT is meaningless or the same thing as SIMD. > NVidia was calling SIMT back in 2010, long before per-thread instruction counters were available in Pascal (GTX 10xx series of GPUs) Right; I'm saying the "SIMT" of 10 years ago has to do with masking and maybe other per-thread control (even though the Connection Machines might have had it as well), and that per-thread counters are now going beyond what we used to know as SIMT, perhaps deserving of some other acronym. Or maybe SIMT is becoming more appropriate and more differentiated from SIMD over time? Is it possible that CM5 should be called SIMT and we simply didn't have that acronym at the time?

darknoon 6y ago

Not even SIMD on recent GPUs, it's SIMT.

Jasper_ 6y ago
SIMT has always just a fancy disguise on top of SIMD hardware, but don't tell the GPU manufacturers I said that :)
dragontamer 6y ago
No one has ever been able to tell me the difference between SIMT and 1980s style SIMD on the Connection Machine 5 (CM-5). Researchers like Blelloch have been writing in SIMD-parallel style "parallel Lisp" or "C-Star" for decades before CUDA. https://www.cs.cmu.edu/~guyb/papers/Ble90.pdf https://www.cs.cmu.edu/~guyb/papers/Ble90.pdf
- dahart 6y ago
  Thread masking is one of the big differences - threads in a warp today can be enabled or disabled independently to enable limited amounts of divergent flow control, and conversely super cheap syncronization. I don't know what the CM5 did - was it limited to always-on sets of vector instructions? Current GPUs also support certain amounts of fully independent thread scheduling, e.g., per-thread instruction counters, etc. Maybe that is beginning to leak out of the SIMT model.
  - dragontamer 6y ago
    > I don't know what the CM5 did - was it limited to always-on sets of vector instructions? EDIT: Woops, I mean CM2. CM5 was MIMD and a slightly different architecture. I've edited this post to say CM-2 instead, but my previous post about CM5 will remain in error. Individual thread masking was available on CM-2, for all 4096 cores that were executing in SIMD-parallel. CM2 was a 4096 x 1-bit SIMD processor. Very limited compared to modern GPUs, but the execution model that CM2 experimented with eventually became the modern GPU architecture. Yes, you can have effective execution masks even with only 1-bit cores. EDIT: See the C-Star manuals, a "per-thread if statement" required a "where" construct instead of if. But yes, the "where" statement performed similarly to "CUDA if": http://people.csail.mit.edu/bradley/cm5docs/AReferenceDescriptionoftheCStarLanguage.pdf http://people.csail.mit.edu/bradley/cm5docs/AReferenceDescri... The MIMD thing that CM-5 did didn't seem to stick. CM-2's SIMD execution seems to be a bigger influence. But C-Star ran on both... and it was the high-level languages like C-Star or Parallel-Lisp that influenced GPU languages like CUDA. > The where statement is involved with setting the context, a process known as contextualization. The context is a parallel boolean mask (i.e., each element of it is true or false) that controls the execution of parallel operations position by position. A different context is associated with each shape object, and the context associated with the current shape is always applied to operators. A CUDA-block was roughly equivalent to a CStar-Shape. A CUDA-if is roughly equivalent to a CStar-where. > Current GPUs also support certain amounts of fully independent thread scheduling, e.g., per-thread instruction counters, etc. Maybe that is beginning to leak out of the SIMT model. NVidia was calling SIMT back in 2010, long before per-thread instruction counters were available in Pascal (GTX 10xx series of GPUs)
    - dahart 6y ago
      Thanks for the C-star link. At first glance shapes look similar to what have become launch dimensions. I think it'd be pretty interesting to hear a comparison from someone who's used both C-star and CUDA, about the plusses and minuses of each... have you used both? > Very limited compared to modern GPUs Yeah, I'd expect the real true differences between CM5 SIMD and Pascal SIMT is just buried in the subtleties of those limitations compared to today's hardware, and that fundamentally they aren't super different ideas. In my mind, that doesn't mean that SIMT is meaningless or the same thing as SIMD. > NVidia was calling SIMT back in 2010, long before per-thread instruction counters were available in Pascal (GTX 10xx series of GPUs) Right; I'm saying the "SIMT" of 10 years ago has to do with masking and maybe other per-thread control (even though the Connection Machines might have had it as well), and that per-thread counters are now going beyond what we used to know as SIMT, perhaps deserving of some other acronym. Or maybe SIMT is becoming more appropriate and more differentiated from SIMD over time? Is it possible that CM5 should be called SIMT and we simply didn't have that acronym at the time?