4 ms·

That doesn't make much sense to me. SIMD is a parallelization technique.

jlokier 6y ago
I think they mean that SIMD only provides a very particular, limited kind of parallelism. It's only useful for some kinds of parallel problems. Anything where the execution traces diverge significantly requires more execution cores, to sequence them separately. For example serving multiple web pages in parallel. Anything where the executation traces are identical or near-identical across many repetitions in lockstep, is fine with one execution core and SIMD ALUs. For example dense or block matrix arithmetic. Modern GPUs are a hybrid of both ideas. Shared execution cores and SIMD ALUs because of high repetition of identical logic, but still many execution cores (but fewer than ALUs) so there is some amount of decoupled parallelism as well, running different tasks at the same time.
- cogman10 6y ago
  GPUs are somewhat more like "Super powerful SIMD". That is, you tell a GPU "I want n cores to run this block of code" and that's just what it will do. SIMD is much more primitive. It is more along the scale of "Multiple these 4 numbers by 5" and that's it. A GPU kernel can have branching, conditionals, etc. SIMD is limited to one operation on a small set of data (512 bytes for AVX-512).
  - jlokier 6y ago
    CPU SIMD can do conditionals and branches in the same way as GPUs, using per-lane conditionals. A modern GPU isn't limited to running one block of code at a time (on n cores). It will schedule a few different blocks of code independently - up to the number of execution units on the GPU, multiplied by the number of latency-hiding time slots per execution unit.
GordonS 6y ago
Yes, but not the kind of parallelization technique most think of (multi-threading). SIMD is still very useful elsewhere of course, but there is a lot of software for which it's not very relevant.
PaulDavisThe1st 6y ago
SIMD => Single Instruction Multiple Data works great if you have, literally, one instruction you want to apply to multiple data. not relevant if "parallelism" means "thread 1 is doing foo, thread 2 is doing bar, and thread 3 is doing baz".

I think they mean that SIMD only provides a very particular, limited kind of parallelism. It's only useful for some kinds of parallel problems. Anything where the execution traces diverge significantly requires more execution cores, to sequence them separately. For example serving multiple web pages in parallel. Anything where the executation traces are identical or near-identical across many repetitions in lockstep, is fine with one execution core and SIMD ALUs. For example dense or block matrix arithmetic. Modern GPUs are a hybrid of both ideas. Shared execution cores and SIMD ALUs because of high repetition of identical logic, but still many execution cores (but fewer than ALUs) so there is some amount of decoupled parallelism as well, running different tasks at the same time.

cogman10 6y ago
GPUs are somewhat more like "Super powerful SIMD". That is, you tell a GPU "I want n cores to run this block of code" and that's just what it will do. SIMD is much more primitive. It is more along the scale of "Multiple these 4 numbers by 5" and that's it. A GPU kernel can have branching, conditionals, etc. SIMD is limited to one operation on a small set of data (512 bytes for AVX-512).
- jlokier 6y ago
  CPU SIMD can do conditionals and branches in the same way as GPUs, using per-lane conditionals. A modern GPU isn't limited to running one block of code at a time (on n cores). It will schedule a few different blocks of code independently - up to the number of execution units on the GPU, multiplied by the number of latency-hiding time slots per execution unit.

GordonS 6y ago

Yes, but not the kind of parallelization technique most think of (multi-threading). SIMD is still very useful elsewhere of course, but there is a lot of software for which it's not very relevant.

PaulDavisThe1st 6y ago

SIMD => Single Instruction Multiple Data works great if you have, literally, one instruction you want to apply to multiple data. not relevant if "parallelism" means "thread 1 is doing foo, thread 2 is doing bar, and thread 3 is doing baz".