4 ms·

I believe the concept says to focus more on task-based parallelism rather than data-based parallelism. In Go, it is easy to create multiple tasks/workers each

by dangets 14y ago

I believe the concept says to focus more on task-based parallelism rather than data-based parallelism. In Go, it is easy to create multiple tasks/workers each with a different job. This is implicitly parallelizable - each task can (but doesn't have to) work within their own thread. The only time when the workers can't run in parallel is when they are waiting on communication from another worker or outside process. This is opposed to data level parallelism where each thread is doing the nearly exactly same instructions on different input, with little to no communication between the threads. An example would be to increase the blue level on each pixel in an image. Each pixel can be operated on individually and be performed in parallel. So - the push is for more task-based parallelism in programs. It is very flexible in that it can run actually in parallel or sequentially and it won't matter on the outcome of the program.

pcwalton 14y ago
I don't agree with that; task parallelism is easier than data parallelism in an actor/CSP-based system, but both have their place. Take something like x264 -- task parallelism will not help it, unless you're encoding multiple videos at one time. But data parallelism (SIMD, in particular) is the reason it's the fastest encoder around.
- astrange 14y ago
  x264 gets equal use from threads per-frame as it does SIMD per-pixel. There's a pretty much linear speed increase for each new thread, even.
  - pcwalton 14y ago
    Interesting, I didn't know that video encoding could be parallelized per-frame in that way. That's really cool; I stand corrected. Still, I think it's clear that an x264 with N threads per frame with no per-pixel SIMD would lose to the current x264 with N threads per frame. The key is that x264 is making good use of task parallelism and data parallelism.
- KirinDave 14y ago
  In general, task paralellism can model anything data parallelism can and given "a sufficiently smart compiler" you can end up with the same result. This is an informal corollary of Needham's Duality (which is itself informal, so make of it what you will). Our current hardware tends to offer great data parallelism for homogenous task queues and task parallelism for heterogenous task queues. Given that, we task parallelism needs a lot more consideration from a human. It's also the case that our current implementations of data parallelism tend to focus on shared memory computations, so their scope is a lot more limited than the distributed-system-conflated discipline of task-oriented concurrency, where we're currently having an explosion of engineering.
  - pcwalton 14y ago
    For a "sufficiently smart compiler" you mean autovectorization. That's a hard problem. I've mentioned before that I don't think SIMD is going away anytime soon. It has so many upsides (cache locality, simple implementation in hardware due to the single-instruction nature of it) that I think designs that don't take advantage of it will always be at a disadvantage for the foreseeable future.

by dangets 14y ago

pcwalton 14y ago

I don't agree with that; task parallelism is easier than data parallelism in an actor/CSP-based system, but both have their place. Take something like x264 -- task parallelism will not help it, unless you're encoding multiple videos at one time. But data parallelism (SIMD, in particular) is the reason it's the fastest encoder around.

astrange 14y ago
x264 gets equal use from threads per-frame as it does SIMD per-pixel. There's a pretty much linear speed increase for each new thread, even.
- pcwalton 14y ago
  Interesting, I didn't know that video encoding could be parallelized per-frame in that way. That's really cool; I stand corrected. Still, I think it's clear that an x264 with N threads per frame with no per-pixel SIMD would lose to the current x264 with N threads per frame. The key is that x264 is making good use of task parallelism and data parallelism.
KirinDave 14y ago
In general, task paralellism can model anything data parallelism can and given "a sufficiently smart compiler" you can end up with the same result. This is an informal corollary of Needham's Duality (which is itself informal, so make of it what you will). Our current hardware tends to offer great data parallelism for homogenous task queues and task parallelism for heterogenous task queues. Given that, we task parallelism needs a lot more consideration from a human. It's also the case that our current implementations of data parallelism tend to focus on shared memory computations, so their scope is a lot more limited than the distributed-system-conflated discipline of task-oriented concurrency, where we're currently having an explosion of engineering.
- pcwalton 14y ago
  For a "sufficiently smart compiler" you mean autovectorization. That's a hard problem. I've mentioned before that I don't think SIMD is going away anytime soon. It has so many upsides (cache locality, simple implementation in hardware due to the single-instruction nature of it) that I think designs that don't take advantage of it will always be at a disadvantage for the foreseeable future.