5 ms·

Parallel Programming for C and C++ Done Right

90 points by timClicks 14y ago

ternaryoperator 14y ago
Cilk/Cilk+ is not the answer, despite years of Intel promoting it and open-sourcing it. Intel has touted many other || technologies in the past: OpenMP, TBB, and now Cilk+. These are all useful tools, but none of them is the way of boldly moving forward with || programming, IMHO. I believe the easier || programming will come from actors, CSPs, channels and other technologies that provide safe concurrency as the working basis.
- ryanmolden 14y ago
  Did you mean CPS instead of CSP? Not trying to be snarky/pedantic, just making sure there isn't some acronym in this space I am unfamiliar with (not entirely unlikely :)). Edit: nope, you probably meant CSPs (http://en.wikipedia.org/wiki/Communicating_sequential_processes http://en.wikipedia.org/wiki/Communicating_sequential_proces...), sorry.
- comex 14y ago
  True, Concurrency is not Parallelism [http://concur.rspace.googlecode.com/hg/talk/concur.html http://concur.rspace.googlecode.com/hg/talk/concur.html], but sometimes all you need is parallel execution of a single algorithm, and you need a set of primitives that succinctly and efficiently express that parallelism... not to mention integrated access to SIMD and GPUs, where only parallelism need apply. I'd say there needs to be both a high level "break your app's work into a heterogeneous set of processes" API, like CSP, and a low level "run an algorithm on this array really fast" API.
- deleted 14y ago
  [deleted]
- gcp 14y ago
  Cilk/Cilk+ is not the answer, despite years of Intel promoting it and open-sourcing it. Intel removed the features that gave Cilk a reason to exist: inlets and aborts. They're useful for (partially) parallelizing hard-to-parallelize problems that don't fit well into the other frameworks like OpenMP etc. Why did they remove them? I'm guessing because they were difficult to do well, and the algorithms that need them don't present such nice linear scaling graphs for marketing slides. However, without those features, Cilk just doesn't distinguish itself enough from OpenCL, OpenMP etc. Parallelizing easy to parallelize problems isn't the problem, it's the others we need help in dealing with! The poster childs/demos for the original Cilk were parallelized chessprograms, some of which did quite well in real tournaments. It's a very well studied area that exhibits a lot of parallelism, but not in a form that's easy to extract (hence there are no competitive ones using OpenMP, GPUs etc). Cilk was able to do it, a major achievement. But you can't even construct those in Intel's crippled Cilk version any more. Well, not any that would be competitive, anyway, which is the point to begin with. If your solution only solves the easy problems others have already solved, what exactly is your right of existence?
positr0n 14y ago
I'd love to get in to a part of the industry where people care about stuff like this (purely for selfish reasons... web development is fun too but the barrier to entry is a lot lower). Can anyone comment on the number and quality of "hard core" C++ jobs and what you think the trajectory will be like? Right now I use C++ almost exclusively at work but the code/concepts involved aren't too difficult.
- jandrewrogers 14y ago
  C++ is increasingly used for high-performance and massively parallel systems. While I used to work on systems primarily written in Java (large-scale analytics and databases) everything new is being done in C++, especially any kind of high-end compute environments. C++ has a few very real advantages on modern architectures and when tackling modern problems. With C++11, it is also a pretty decent programming language in terms of expressiveness. C++ has two big advantages over alternatives like Java: very low and deterministic processing latency (important for real-time) and very efficient memory model (important for throughput). These are really both about memory, and C++ gives you detailed control in a way few other languages do. As to why this is important, memory performance has been scaling more slowly than many other aspects of computing architecture such that it is the bottleneck for a growing number of applications. C++ allows you to be very efficient with the memory architecture without much effort. If your application is fundamentally bound by memory performance, competent C++ can get you 2-10x returns on performance in real systems relative to languages like Java. (For some other tight-loop, CPU-bound codes, not so much.) I'm pretty bull-ish on C++11. It is not so much that I am a fan of the language but that I am a fan of what it can do in terms of performance for databases and large-scale systems. Most of the high-scale and high-performance development going forward seems to be targeted at C++ these days. That was not always the case but the requirements of modern applications are somewhat forcing that choice.
  - qznc 14y ago
    If you want parallel/concurrent programming there are better languages than C++. However, additional requirements (real-time, performance, size, ...) make C++ desirable again, since C++ tries very hard to avoid tradeoffs in these terms.
tmurray 14y ago
Foreword: I'm biased, as I worked on CUDA for several years. The conclusions offered by this deck are mostly FUD. First of all, Haswell, the architecture where those transactional memory primitives are available, isn't out for another year. Saying Knights Corner was available on November 2011 is also deceptive; Intel demo'd it at SC11, but you can't buy one yet. Second, he helpfully glosses over that Cilk++'s elemental functions are identical to how CUDA and ISPC work; write a specially decorated single-threaded function, use a specially decorated function call, and end up with parallel work. I think it's exceedingly likely that the industry will standardize on this as the data-parallel methodology of choice within the next ten years. That timeframe will depend on how quickly GPUs and CPUs converge in terms of functionality (with vastly different performance characteristics). Task-parallel stuff will be done with something else. The really difficult question will be how to get performance portability. C++ (or Fortran) code that runs well on Haswell will probably run like crap on KNC and vice-versa due to differences in the number of threads you need in flight, cache sizes, vast latency differences, etc. (Look at OpenCL running on two GPUs or especially CPU vs GPU as an example today.) Solving that is going to be the real challenge.
patrikmcguire 14y ago
Has anyone had any real-world experience using X!0? http://en.wikipedia.org/wiki/X10_(programming_language) http://en.wikipedia.org/wiki/X10_(programming_language) It came out of IBM (Eclipse license) at about the same time as Watson, and I've gotten the impression that it (compiled down to C++) was the main language used for it. One of the authors was a guest professor whenever I took my parallel programming course and wound up teaching about half the classes, so its abilities and use may have been exaggerated slightly, but it has a lot of constructs built in that I'd imagine to be terrible to implement otherwise - good globally synchronized clocks and memory management across everything on the current "place" (roughly one physical computer), although you could still had to manage memory you sent to different places manually. But Wikipedia says Watson's built mostly on Hadoop, where the coolest features wouldn't really have much of an effect, so it may be just a crazy research language. I was just curious.
- qznc 14y ago
  X10 is a research language: Interesting concepts, crappy implementation. X10 is also dying as IBM is shutting down its research groups. Unless some other institution (some university chair?) will step up for maintenance, its development will halt soon.
  - scott_s 14y ago
    IBM is shutting down its research groups That is not a true statement.
- scott_s 14y ago
  Watson is not "built on" Hadoop. From http://www.aaai.org/ojs/index.php/aimagazine/article/view/2303/ http://www.aaai.org/ojs/index.php/aimagazine/article/view/23...: To preprocess the corpus and create fast run-time indices we used Hadoop. UIMA annotators were easily deployed as mappers in the Hadoop map-reduce framework. Hadoop distributes the content over the cluster to afford high CPU utilization and provides convenient tools for deploying, managing, and monitoring the corpus analysis process. The online parts of Watson use UIMA: http://uima.apache.org/ http://uima.apache.org/
jmpeax 14y ago
Lots of (R)s and TMs in this Intel TBB advertisement.
- trekkin 14y ago
  TBB is OSS, and we use parts of it (atomics) daily, pretty much as advertised.
bjornsing 14y ago
The first part of the title is descriptive, but there's not much of anything "Done Right" in there that I can see. Just the same old.

90 points by timClicks 14y ago

Cilk/Cilk+ is not the answer, despite years of Intel promoting it and open-sourcing it. Intel has touted many other || technologies in the past: OpenMP, TBB, and now Cilk+. These are all useful tools, but none of them is the way of boldly moving forward with || programming, IMHO. I believe the easier || programming will come from actors, CSPs, channels and other technologies that provide safe concurrency as the working basis.

ryanmolden 14y ago
Did you mean CPS instead of CSP? Not trying to be snarky/pedantic, just making sure there isn't some acronym in this space I am unfamiliar with (not entirely unlikely :)). Edit: nope, you probably meant CSPs (http://en.wikipedia.org/wiki/Communicating_sequential_processes http://en.wikipedia.org/wiki/Communicating_sequential_proces...), sorry.
comex 14y ago
True, Concurrency is not Parallelism [http://concur.rspace.googlecode.com/hg/talk/concur.html http://concur.rspace.googlecode.com/hg/talk/concur.html], but sometimes all you need is parallel execution of a single algorithm, and you need a set of primitives that succinctly and efficiently express that parallelism... not to mention integrated access to SIMD and GPUs, where only parallelism need apply. I'd say there needs to be both a high level "break your app's work into a heterogeneous set of processes" API, like CSP, and a low level "run an algorithm on this array really fast" API.
deleted 14y ago
[deleted]
gcp 14y ago
Cilk/Cilk+ is not the answer, despite years of Intel promoting it and open-sourcing it. Intel removed the features that gave Cilk a reason to exist: inlets and aborts. They're useful for (partially) parallelizing hard-to-parallelize problems that don't fit well into the other frameworks like OpenMP etc. Why did they remove them? I'm guessing because they were difficult to do well, and the algorithms that need them don't present such nice linear scaling graphs for marketing slides. However, without those features, Cilk just doesn't distinguish itself enough from OpenCL, OpenMP etc. Parallelizing easy to parallelize problems isn't the problem, it's the others we need help in dealing with! The poster childs/demos for the original Cilk were parallelized chessprograms, some of which did quite well in real tournaments. It's a very well studied area that exhibits a lot of parallelism, but not in a form that's easy to extract (hence there are no competitive ones using OpenMP, GPUs etc). Cilk was able to do it, a major achievement. But you can't even construct those in Intel's crippled Cilk version any more. Well, not any that would be competitive, anyway, which is the point to begin with. If your solution only solves the easy problems others have already solved, what exactly is your right of existence?

positr0n 14y ago

I'd love to get in to a part of the industry where people care about stuff like this (purely for selfish reasons... web development is fun too but the barrier to entry is a lot lower). Can anyone comment on the number and quality of "hard core" C++ jobs and what you think the trajectory will be like? Right now I use C++ almost exclusively at work but the code/concepts involved aren't too difficult.

jandrewrogers 14y ago
C++ is increasingly used for high-performance and massively parallel systems. While I used to work on systems primarily written in Java (large-scale analytics and databases) everything new is being done in C++, especially any kind of high-end compute environments. C++ has a few very real advantages on modern architectures and when tackling modern problems. With C++11, it is also a pretty decent programming language in terms of expressiveness. C++ has two big advantages over alternatives like Java: very low and deterministic processing latency (important for real-time) and very efficient memory model (important for throughput). These are really both about memory, and C++ gives you detailed control in a way few other languages do. As to why this is important, memory performance has been scaling more slowly than many other aspects of computing architecture such that it is the bottleneck for a growing number of applications. C++ allows you to be very efficient with the memory architecture without much effort. If your application is fundamentally bound by memory performance, competent C++ can get you 2-10x returns on performance in real systems relative to languages like Java. (For some other tight-loop, CPU-bound codes, not so much.) I'm pretty bull-ish on C++11. It is not so much that I am a fan of the language but that I am a fan of what it can do in terms of performance for databases and large-scale systems. Most of the high-scale and high-performance development going forward seems to be targeted at C++ these days. That was not always the case but the requirements of modern applications are somewhat forcing that choice.
- qznc 14y ago
  If you want parallel/concurrent programming there are better languages than C++. However, additional requirements (real-time, performance, size, ...) make C++ desirable again, since C++ tries very hard to avoid tradeoffs in these terms.

tmurray 14y ago

Foreword: I'm biased, as I worked on CUDA for several years. The conclusions offered by this deck are mostly FUD. First of all, Haswell, the architecture where those transactional memory primitives are available, isn't out for another year. Saying Knights Corner was available on November 2011 is also deceptive; Intel demo'd it at SC11, but you can't buy one yet. Second, he helpfully glosses over that Cilk++'s elemental functions are identical to how CUDA and ISPC work; write a specially decorated single-threaded function, use a specially decorated function call, and end up with parallel work. I think it's exceedingly likely that the industry will standardize on this as the data-parallel methodology of choice within the next ten years. That timeframe will depend on how quickly GPUs and CPUs converge in terms of functionality (with vastly different performance characteristics). Task-parallel stuff will be done with something else. The really difficult question will be how to get performance portability. C++ (or Fortran) code that runs well on Haswell will probably run like crap on KNC and vice-versa due to differences in the number of threads you need in flight, cache sizes, vast latency differences, etc. (Look at OpenCL running on two GPUs or especially CPU vs GPU as an example today.) Solving that is going to be the real challenge.

patrikmcguire 14y ago

Has anyone had any real-world experience using X!0? http://en.wikipedia.org/wiki/X10_(programming_language) http://en.wikipedia.org/wiki/X10_(programming_language) It came out of IBM (Eclipse license) at about the same time as Watson, and I've gotten the impression that it (compiled down to C++) was the main language used for it. One of the authors was a guest professor whenever I took my parallel programming course and wound up teaching about half the classes, so its abilities and use may have been exaggerated slightly, but it has a lot of constructs built in that I'd imagine to be terrible to implement otherwise - good globally synchronized clocks and memory management across everything on the current "place" (roughly one physical computer), although you could still had to manage memory you sent to different places manually. But Wikipedia says Watson's built mostly on Hadoop, where the coolest features wouldn't really have much of an effect, so it may be just a crazy research language. I was just curious.

qznc 14y ago
X10 is a research language: Interesting concepts, crappy implementation. X10 is also dying as IBM is shutting down its research groups. Unless some other institution (some university chair?) will step up for maintenance, its development will halt soon.
- scott_s 14y ago
  IBM is shutting down its research groups That is not a true statement.
scott_s 14y ago
Watson is not "built on" Hadoop. From http://www.aaai.org/ojs/index.php/aimagazine/article/view/2303/ http://www.aaai.org/ojs/index.php/aimagazine/article/view/23...: To preprocess the corpus and create fast run-time indices we used Hadoop. UIMA annotators were easily deployed as mappers in the Hadoop map-reduce framework. Hadoop distributes the content over the cluster to afford high CPU utilization and provides convenient tools for deploying, managing, and monitoring the corpus analysis process. The online parts of Watson use UIMA: http://uima.apache.org/ http://uima.apache.org/

jmpeax 14y ago

Lots of (R)s and TMs in this Intel TBB advertisement.

trekkin 14y ago
TBB is OSS, and we use parts of it (atomics) daily, pretty much as advertised.

bjornsing 14y ago

The first part of the title is descriptive, but there's not much of anything "Done Right" in there that I can see. Just the same old.