5 ms·

I write distributed-memory software and that was my first response when I first saw this a few years ago. But we are getting more on-node parallelism (and furt

I write distributed-memory software and that was my first response when I first saw this a few years ago. But we are getting more on-node parallelism (and further on-node parallelism is where most of the next 10 years of performance is expected to come) so these techniques are more relevant to our large-scale computing than you'd otherwise think. Especially as we push strong-scaling limits and attempt to coordinate threads in order to share caches, the standard application of domain decomposition entails higher overheads than we would like to pay.

deletes 13y ago
I have written a pretty simple job-worker system in c ,and I'm interested in this field. Do you have any recommendations for books/resources?
- gtani 13y ago
  For cuda books, you can go thru Wilt's, it's pretty dense https://twitter.com/CUDAHandbook https://twitter.com/CUDAHandbook There's a handful of papers on attacking the 13 (formerly 8) dwarf problems http://tom.scogland.com/pubs/pdf/feng-ocd-icpe12.pdf http://tom.scogland.com/pubs/pdf/feng-ocd-icpe12.pdf I don't have any specifics on Kaveri memory model but there's lots of small reviews: http://www.tomshardware.com/reviews/a10-7850k-a8-7600-kaveri,3725-3.html http://www.tomshardware.com/reviews/a10-7850k-a8-7600-kaveri... compiler research and thread debugging: http://iacoma.cs.uiuc.edu/iacoma-papers/Illinois_parallelism_book.pdf http://iacoma.cs.uiuc.edu/iacoma-papers/Illinois_parallelism...
  - gtani 13y ago
    and Eijkhout's book, excellent place to start, (but web server down at the moment) http://tacc-web.austin.utexas.edu/veijkhout/public_html/Articles/EijkhoutIntroToHPC.pdf http://tacc-web.austin.utexas.edu/veijkhout/public_html/Arti... Introduction to High Performance Scientific Computing - Victor Eijkhout with Edmond Chow, Robert van de Geijn
deleted 13y ago
[deleted]
seanmcdirmid 13y ago
The question is even fuzzier: how much of that will be massive data parallelism via an on-node GPU? We are already reaching a point where a single machine with 4 CUDA cards can out perform a distributed cluster of 1000 nodes on some applications (the trick would be to eventually do both). The memory hierarchy is our enemy here: the reason GPUs have done so well is that they schedule memory just as much (if not more) as they do computation. If you are going to go through the trouble of coordinating threads to share caches (and this is possible at all), you might have a GPU-friendly problem.
- pbsd 13y ago
  Can you point to a problem where a GPU actually outperforms a CPU by 250x and the CPU is not being criminally underused? I have tried to find such examples, and never did. Unless, maybe, you mean the communication costs are the real bottleneck in such cases? In which case I don't see the relevance of the GPU angle.
  - seanmcdirmid 13y ago
    Yes, distributed clusters are limited by communicate costs, not computation power. Parallel computing is in general limited by communication costs, even on one node (as the amount of time it takes to service a cache miss). Minimizing communication is important in both cases. DNN training is one problem where the GPU solution vastly outperforms the distributed HPC solution.
  - adrianN 13y ago
    GPUs are useful for algebraic geometry. The speedup is not quite 250, but something like 80 or so. http://www.mpi-inf.mpg.de/~emeliyan/phd_thesis.pdf http://www.mpi-inf.mpg.de/~emeliyan/phd_thesis.pdf
  - fulafel 13y ago
    Switching one of the more fancier looking PS4 titles over to software rendering might do it. :)
- jedbrown 13y ago

deletes 13y ago

I have written a pretty simple job-worker system in c ,and I'm interested in this field. Do you have any recommendations for books/resources?

gtani 13y ago
For cuda books, you can go thru Wilt's, it's pretty dense https://twitter.com/CUDAHandbook https://twitter.com/CUDAHandbook There's a handful of papers on attacking the 13 (formerly 8) dwarf problems http://tom.scogland.com/pubs/pdf/feng-ocd-icpe12.pdf http://tom.scogland.com/pubs/pdf/feng-ocd-icpe12.pdf I don't have any specifics on Kaveri memory model but there's lots of small reviews: http://www.tomshardware.com/reviews/a10-7850k-a8-7600-kaveri,3725-3.html http://www.tomshardware.com/reviews/a10-7850k-a8-7600-kaveri... compiler research and thread debugging: http://iacoma.cs.uiuc.edu/iacoma-papers/Illinois_parallelism_book.pdf http://iacoma.cs.uiuc.edu/iacoma-papers/Illinois_parallelism...
- gtani 13y ago
  and Eijkhout's book, excellent place to start, (but web server down at the moment) http://tacc-web.austin.utexas.edu/veijkhout/public_html/Articles/EijkhoutIntroToHPC.pdf http://tacc-web.austin.utexas.edu/veijkhout/public_html/Arti... Introduction to High Performance Scientific Computing - Victor Eijkhout with Edmond Chow, Robert van de Geijn

deleted 13y ago

[deleted]

seanmcdirmid 13y ago

The question is even fuzzier: how much of that will be massive data parallelism via an on-node GPU? We are already reaching a point where a single machine with 4 CUDA cards can out perform a distributed cluster of 1000 nodes on some applications (the trick would be to eventually do both). The memory hierarchy is our enemy here: the reason GPUs have done so well is that they schedule memory just as much (if not more) as they do computation. If you are going to go through the trouble of coordinating threads to share caches (and this is possible at all), you might have a GPU-friendly problem.

pbsd 13y ago
Can you point to a problem where a GPU actually outperforms a CPU by 250x and the CPU is not being criminally underused? I have tried to find such examples, and never did. Unless, maybe, you mean the communication costs are the real bottleneck in such cases? In which case I don't see the relevance of the GPU angle.
- seanmcdirmid 13y ago
  Yes, distributed clusters are limited by communicate costs, not computation power. Parallel computing is in general limited by communication costs, even on one node (as the amount of time it takes to service a cache miss). Minimizing communication is important in both cases. DNN training is one problem where the GPU solution vastly outperforms the distributed HPC solution.
- adrianN 13y ago
  GPUs are useful for algebraic geometry. The speedup is not quite 250, but something like 80 or so. http://www.mpi-inf.mpg.de/~emeliyan/phd_thesis.pdf http://www.mpi-inf.mpg.de/~emeliyan/phd_thesis.pdf
- fulafel 13y ago
  Switching one of the more fancier looking PS4 titles over to software rendering might do it. :)
jedbrown 13y ago