4 ms·
To clarify something not evident in the title, but which became obvious when I looked through the doc, this is about small-scale parallelism. The kind you find
by jandrewrogers 13y ago
To clarify something not evident in the title, but which became obvious when I looked through the doc, this is about small-scale parallelism. The kind you find in a single (non-exotic) machine.
Massive-scale parallelism uses different data structures, algorithms, and programming models not discussed.
- jedbrown 13y agoI write distributed-memory software and that was my first response when I first saw this a few years ago. But we are getting more on-node parallelism (and further on-node parallelism is where most of the next 10 years of performance is expected to come) so these techniques are more relevant to our large-scale computing than you'd otherwise think. Especially as we push strong-scaling limits and attempt to coordinate threads in order to share caches, the standard application of domain decomposition entails higher overheads than we would like to pay.
- deletes 13y agoI have written a pretty simple job-worker system in c ,and I'm interested in this field. Do you have any recommendations for books/resources?
- gtani 13y agoFor cuda books, you can go thru Wilt's, it's pretty dense https://twitter.com/CUDAHandbook https://twitter.com/CUDAHandbook There's a handful of papers on attacking the 13 (formerly 8) dwarf problems http://tom.scogland.com/pubs/pdf/feng-ocd-icpe12.pdf http://tom.scogland.com/pubs/pdf/feng-ocd-icpe12.pdf I don't have any specifics on Kaveri memory model but there's lots of small reviews: http://www.tomshardware.com/reviews/a10-7850k-a8-7600-kaveri,3725-3.html http://www.tomshardware.com/reviews/a10-7850k-a8-7600-kaveri... compiler research and thread debugging: http://iacoma.cs.uiuc.edu/iacoma-papers/Illinois_parallelism_book.pdf http://iacoma.cs.uiuc.edu/iacoma-papers/Illinois_parallelism...
- gtani 13y agoand Eijkhout's book, excellent place to start, (but web server down at the moment) http://tacc-web.austin.utexas.edu/veijkhout/public_html/Articles/EijkhoutIntroToHPC.pdf http://tacc-web.austin.utexas.edu/veijkhout/public_html/Arti... Introduction to High Performance Scientific Computing - Victor Eijkhout with Edmond Chow, Robert van de Geijn
- deleted 13y ago[deleted]
- seanmcdirmid 13y agoThe question is even fuzzier: how much of that will be massive data parallelism via an on-node GPU? We are already reaching a point where a single machine with 4 CUDA cards can out perform a distributed cluster of 1000 nodes on some applications (the trick would be to eventually do both). The memory hierarchy is our enemy here: the reason GPUs have done so well is that they schedule memory just as much (if not more) as they do computation. If you are going to go through the trouble of coordinating threads to share caches (and this is possible at all), you might have a GPU-friendly problem.
- pbsd 13y agoCan you point to a problem where a GPU actually outperforms a CPU by 250x and the CPU is not being criminally underused? I have tried to find such examples, and never did. Unless, maybe, you mean the communication costs are the real bottleneck in such cases? In which case I don't see the relevance of the GPU angle.
- seanmcdirmid 13y agoYes, distributed clusters are limited by communicate costs, not computation power. Parallel computing is in general limited by communication costs, even on one node (as the amount of time it takes to service a cache miss). Minimizing communication is important in both cases. DNN training is one problem where the GPU solution vastly outperforms the distributed HPC solution.
- adrianN 13y agoGPUs are useful for algebraic geometry. The speedup is not quite 250, but something like 80 or so. http://www.mpi-inf.mpg.de/~emeliyan/phd_thesis.pdf http://www.mpi-inf.mpg.de/~emeliyan/phd_thesis.pdf
- fulafel 13y agoSwitching one of the more fancier looking PS4 titles over to software rendering might do it. :)
- jedbrown 13y ago
- LnxPrgr3 13y agoI can tell I live in a different world—I immediately thought of multi-threaded programming on a single machine. Modern individual machines give us all kinds of opportunity for parallel execution—even my phone has vector instructions and two CPU cores. Modern Intel CPUs have 256-bit wide vectors, meaning you can do 8 floating point operations in a single instruction, assuming you can phrase your problem in the right way. If that's not enough, you can throw a pile of these cores at the problem, assuming you can coordinate work between these threads. My experience has been most programmers struggle hard to build multi-threaded applications, countless hours lost to tracking down race conditions, deadlocks, and unexpectedly bad performance, which is a shame. Scanning through this book, it doesn't appear to cover vector instructions, but it does look like it covers coordinating multiple threads in amazing depth. I feel like I already have a decent handle on multi-threaded programming, but I will be reading this book, and I'll probably learn plenty.
- shmerl 13y ago> My experience has been most programmers struggle hard to build multi-threaded applications, countless hours lost to tracking down race conditions, deadlocks, and unexpectedly bad performance, which is a shame. A lot of it is caused by tools which are by design prone to such conditions and unless you consciously and meticulously follow very strict guidelines, such problems are inevitable. But some tools try to solve it with different design approach (such as Rust language for example) which prevents many of those potential pitfalls implicitly. I wonder if the book is focused on shared memory and locking only, or covers broader range of methodologies (at the first glance it's mostly about classic shared memory and mutual exclusion approach).