3 ms·

Note that Amdahl's law doesn't apply if you change the entire algorithm (program) to fit better on parallel hardware (i.e. the task is not fixed). From my deal

Note that Amdahl's law doesn't apply if you change the entire algorithm (program) to fit better on parallel hardware (i.e. the task is not fixed). From my dealings with parallel algorithms, there is much less sequential dependency than I think most people expect.

nuc1e0n 3y ago
What day to day problems do people have that require parallel processing to solve anyway? Sure graphics and neural networks, but is having a neural network the end problem that needs solving? Maybe I'm just a grumpy old programmer but I believe the human mind is specialised to think in terms of serial solutions. I think that's no accident as race conditions and other synchronization issues are difficult to avoid in parallel algorithms. That may be a fundamental property of processing in general, even for algorithms some AI may come up with, which remains mostly fictional at this point.
- winwang 3y ago
  Compilation comes to mind. Efficient distributed compute also usually prefers vertical scaling (i.e. more cores per node).
- Legend2440 3y ago
  >I believe the human mind is specialised to think in terms of serial solutions The brain is massively parallel - 80 billion neurons operating asynchronously at only 100hz. >is having a neural network the end problem that needs solving? Absolutely! Neural networks are a general-purpose way to solve a huge number of problems, especially problems that involve raw data or the messy real world.
  - nuc1e0n 3y ago
    I'm talking about the solutions we tend to produce, not our internal workings. You contradicted yourself in that last paragraph. I don't know what else to say to you about that.
    - dgfitz 3y ago
      I know I think in parallel often enough that two threads end up disagreeing with each other and a 3rd one has to sort it out. I can’t begin to describe the internal fork() sensation that happens in my noggin constantly.

nuc1e0n 3y ago

What day to day problems do people have that require parallel processing to solve anyway? Sure graphics and neural networks, but is having a neural network the end problem that needs solving? Maybe I'm just a grumpy old programmer but I believe the human mind is specialised to think in terms of serial solutions. I think that's no accident as race conditions and other synchronization issues are difficult to avoid in parallel algorithms. That may be a fundamental property of processing in general, even for algorithms some AI may come up with, which remains mostly fictional at this point.

winwang 3y ago
Compilation comes to mind. Efficient distributed compute also usually prefers vertical scaling (i.e. more cores per node).
Legend2440 3y ago
>I believe the human mind is specialised to think in terms of serial solutions The brain is massively parallel - 80 billion neurons operating asynchronously at only 100hz. >is having a neural network the end problem that needs solving? Absolutely! Neural networks are a general-purpose way to solve a huge number of problems, especially problems that involve raw data or the messy real world.
- nuc1e0n 3y ago
  I'm talking about the solutions we tend to produce, not our internal workings. You contradicted yourself in that last paragraph. I don't know what else to say to you about that.
  - dgfitz 3y ago
    I know I think in parallel often enough that two threads end up disagreeing with each other and a 3rd one has to sort it out. I can’t begin to describe the internal fork() sensation that happens in my noggin constantly.