5 ms·

The point is that these easy cases may just referred to as "work to do". Using more than one CPU core at the time to process through a backlog of work is just..

by dagss 6y ago

The point is that these easy cases may just referred to as "work to do". Using more than one CPU core at the time to process through a backlog of work is just...the normal case. Doing work simultanously is not what people refer to when they say "parallel programming is hard"; and the pedantery is in insisting that people word themselves to exclude trivially parallel work when what they want to talk about are the cases where parallelism is not trivial. The conversation above is like this: "Solving cubic equations is hard" "Actually, solving x ^ 3 = 27 is easy, just take the cubic root.." "Well that is not what we meant when we said cubic equations, we were interested in the general case..." "I see equations like x^3 = y all the time, it is not that often you have a linear term too..."

dragontamer 6y ago
I see it differently. * "Taking care of your car is hard". * "No its not. 90% of the time, its just changing your oil, refilling the fuel tank, washer fluid, and rotating tires". * "But you have to also replace the transmission fluid and sometimes rebuild the engine" * "True, but those are not a very common tasks". ----------- That's how I see it. Most parallelism is "easy parallelism" in my experience. The goal isn't necessarily to train programmers into being able to do "engine rebuilds", but instead train them for the easy "oil change / rotate tires" part of taking care of cars (or programs). For programming: that's fork / wait(), or pthread_create() / pthread_join(), or #pragma omp parallel for. Those constructs covers the vast, vast majority of parallel constructs I've run across. Yes, mutex is hard. Condition variables are hard. Lock-free programming is hard. High performance (false-sharing / MESI) is hard. GPU ballots / __shared__ memory / butterfly permutes are very hard. Prefix-sum / scan and Mergepath algorithms are hard. Lock-free Producer-consumer is really, really, really hard (Probably PH.D level hard) But... I really don't come across a lot of situations where I need to Mergepath, GPU-ballot, or need to manually program / debug a Lock-free algorithm. That's just the facts. I'm glad I learned these constructs (and I await the day when a high-performance programming problem is given to me such that I need to "pull out the big guns" that I've spent a lot of effort learning)... but I recognize that its not really needed in typical day-to-day programming parallel jobs. Even in GPU-programming: the bulk of GPU-programming is just simple fork/join stuff. A GPU-kernel is a fork, and when the GPU-kernel finishes execution, its an innate join() (with synchronization guaranteed). Its really not that hard in most situations. Schedule your chain of GPU-kernels to execute in order (a "CUDA-stream"), and they automatically join and synchronize with themselves each time the kernel completes. Its that easy in most cases. "Oh, but for maximum performance you want to use dynamic / recursive parallelism, or block-off your data into __shared__ memory and think about L1 caches". Erm, yes, if you care to put that much work into performance its hard. But you're no longer talking about parallelism per se, but instead the very difficult field of high-performance compute at that point.
- dagss 6y ago
  Your analogy is very close to mine. Following your analogy then -- this is a HN thread where car mechanics discuss a book about transmission fluids and engine rebuilding. Did anyone make the point that most programmers often need to care about this stuff?
- imtringued 6y ago
  >* "True, but those are not a very common tasks". Pretty much every survival/simulation game with complex mechanics suffers from low performance because at some point the simulation becomes very CPU intensive but parallelizing the code is still extremely difficult because of extensive interdependencies. If you go with the trivial "divide the problem into the smallest parallelizable components" strategy you will end up with huge interconnected graphs, sometimes there isn't even a second graph that can be put on a second CPU core. If you are insane enough to divide the problem at the agent level then you end up with a synchronization nightmare. Sure, given enough effort you can do these all these things but that only entrenches the perception of parallelism being a last resort instead of being the default solution for everything.

by dagss 6y ago

dragontamer 6y ago

I see it differently. * "Taking care of your car is hard". * "No its not. 90% of the time, its just changing your oil, refilling the fuel tank, washer fluid, and rotating tires". * "But you have to also replace the transmission fluid and sometimes rebuild the engine" * "True, but those are not a very common tasks". ----------- That's how I see it. Most parallelism is "easy parallelism" in my experience. The goal isn't necessarily to train programmers into being able to do "engine rebuilds", but instead train them for the easy "oil change / rotate tires" part of taking care of cars (or programs). For programming: that's fork / wait(), or pthread_create() / pthread_join(), or #pragma omp parallel for. Those constructs covers the vast, vast majority of parallel constructs I've run across. Yes, mutex is hard. Condition variables are hard. Lock-free programming is hard. High performance (false-sharing / MESI) is hard. GPU ballots / __shared__ memory / butterfly permutes are very hard. Prefix-sum / scan and Mergepath algorithms are hard. Lock-free Producer-consumer is really, really, really hard (Probably PH.D level hard) But... I really don't come across a lot of situations where I need to Mergepath, GPU-ballot, or need to manually program / debug a Lock-free algorithm. That's just the facts. I'm glad I learned these constructs (and I await the day when a high-performance programming problem is given to me such that I need to "pull out the big guns" that I've spent a lot of effort learning)... but I recognize that its not really needed in typical day-to-day programming parallel jobs. Even in GPU-programming: the bulk of GPU-programming is just simple fork/join stuff. A GPU-kernel is a fork, and when the GPU-kernel finishes execution, its an innate join() (with synchronization guaranteed). Its really not that hard in most situations. Schedule your chain of GPU-kernels to execute in order (a "CUDA-stream"), and they automatically join and synchronize with themselves each time the kernel completes. Its that easy in most cases. "Oh, but for maximum performance you want to use dynamic / recursive parallelism, or block-off your data into __shared__ memory and think about L1 caches". Erm, yes, if you care to put that much work into performance its hard. But you're no longer talking about parallelism per se, but instead the very difficult field of high-performance compute at that point.

dagss 6y ago
Your analogy is very close to mine. Following your analogy then -- this is a HN thread where car mechanics discuss a book about transmission fluids and engine rebuilding. Did anyone make the point that most programmers often need to care about this stuff?
imtringued 6y ago
>* "True, but those are not a very common tasks". Pretty much every survival/simulation game with complex mechanics suffers from low performance because at some point the simulation becomes very CPU intensive but parallelizing the code is still extremely difficult because of extensive interdependencies. If you go with the trivial "divide the problem into the smallest parallelizable components" strategy you will end up with huge interconnected graphs, sometimes there isn't even a second graph that can be put on a second CPU core. If you are insane enough to divide the problem at the agent level then you end up with a synchronization nightmare. Sure, given enough effort you can do these all these things but that only entrenches the perception of parallelism being a last resort instead of being the default solution for everything.