3 ms·

There are limits to how much programs can be parallelized, see Amdahl's law. Even if Moore's law technically continues, the extra capacity isn't adding the same

There are limits to how much programs can be parallelized, see Amdahl's law. Even if Moore's law technically continues, the extra capacity isn't adding the same amount of extra value to the workflow of end users that it once did. Here's a video that talks about diminishing returns as it relates to graphics in computer games https://m.youtube.com/watch?v=VUCYAws6oW0 https://m.youtube.com/watch?v=VUCYAws6oW0. Diminishing returns in regards to polygon counts themselves have also been discussed: https://www.polygon.com/2013/12/10/5192058/opinion-stop-dwelling-on-graphics-and-embrace-procedural-generation https://www.polygon.com/2013/12/10/5192058/opinion-stop-dwel... Also, even if Moore's law remains in place for now, it will end at some point and the world needs to think about what happens afterward.

winwang 3y ago
Note that Amdahl's law doesn't apply if you change the entire algorithm (program) to fit better on parallel hardware (i.e. the task is not fixed). From my dealings with parallel algorithms, there is much less sequential dependency than I think most people expect.
- nuc1e0n 3y ago
  What day to day problems do people have that require parallel processing to solve anyway? Sure graphics and neural networks, but is having a neural network the end problem that needs solving? Maybe I'm just a grumpy old programmer but I believe the human mind is specialised to think in terms of serial solutions. I think that's no accident as race conditions and other synchronization issues are difficult to avoid in parallel algorithms. That may be a fundamental property of processing in general, even for algorithms some AI may come up with, which remains mostly fictional at this point.
  - winwang 3y ago
    Compilation comes to mind. Efficient distributed compute also usually prefers vertical scaling (i.e. more cores per node).
  - Legend2440 3y ago
    >I believe the human mind is specialised to think in terms of serial solutions The brain is massively parallel - 80 billion neurons operating asynchronously at only 100hz. >is having a neural network the end problem that needs solving? Absolutely! Neural networks are a general-purpose way to solve a huge number of problems, especially problems that involve raw data or the messy real world.
    - nuc1e0n 3y ago
      I'm talking about the solutions we tend to produce, not our internal workings. You contradicted yourself in that last paragraph. I don't know what else to say to you about that.
      - dgfitz 3y ago
        I know I think in parallel often enough that two threads end up disagreeing with each other and a 3rd one has to sort it out. I can’t begin to describe the internal fork() sensation that happens in my noggin constantly.
Legend2440 3y ago
This is true, but parallel programs are not more limited - they're just good at different things. We will write different kinds of programs to do different kinds of tasks because of access to massive parallel compute. Parallelism allows you to do computations that integrate huge amounts of information, like checking every pixel of an image against thousands of weak rules all at once. This is very inefficient to do serially because you can only check one at a time. It's also a huge usecase right now because machine learning lets you learn weak rules from data.
dougmwne 3y ago
It’s funny, because while technically correct, you may end up fundamentally wrong that there was a limit to the value we might find in parallel compute. After all, we may have lie of sight to AGI through parallel scale alone.
- eslaught 3y ago
  What you've identified is known as Gustafson's Law. If you can keep solving larger and larger problems, your solution time doesn't necessarily go down, but "speedup" is potentially uncapped. Of course there may be some practical limit to Gustafson's Law, but I don't think we've found it yet, at least in many scientific domains.
aaron695 3y ago
[dead]

winwang 3y ago

Note that Amdahl's law doesn't apply if you change the entire algorithm (program) to fit better on parallel hardware (i.e. the task is not fixed). From my dealings with parallel algorithms, there is much less sequential dependency than I think most people expect.

nuc1e0n 3y ago
What day to day problems do people have that require parallel processing to solve anyway? Sure graphics and neural networks, but is having a neural network the end problem that needs solving? Maybe I'm just a grumpy old programmer but I believe the human mind is specialised to think in terms of serial solutions. I think that's no accident as race conditions and other synchronization issues are difficult to avoid in parallel algorithms. That may be a fundamental property of processing in general, even for algorithms some AI may come up with, which remains mostly fictional at this point.
- winwang 3y ago
  Compilation comes to mind. Efficient distributed compute also usually prefers vertical scaling (i.e. more cores per node).
- Legend2440 3y ago
  >I believe the human mind is specialised to think in terms of serial solutions The brain is massively parallel - 80 billion neurons operating asynchronously at only 100hz. >is having a neural network the end problem that needs solving? Absolutely! Neural networks are a general-purpose way to solve a huge number of problems, especially problems that involve raw data or the messy real world.
  - nuc1e0n 3y ago
    I'm talking about the solutions we tend to produce, not our internal workings. You contradicted yourself in that last paragraph. I don't know what else to say to you about that.
    - dgfitz 3y ago
      I know I think in parallel often enough that two threads end up disagreeing with each other and a 3rd one has to sort it out. I can’t begin to describe the internal fork() sensation that happens in my noggin constantly.

Legend2440 3y ago

This is true, but parallel programs are not more limited - they're just good at different things. We will write different kinds of programs to do different kinds of tasks because of access to massive parallel compute. Parallelism allows you to do computations that integrate huge amounts of information, like checking every pixel of an image against thousands of weak rules all at once. This is very inefficient to do serially because you can only check one at a time. It's also a huge usecase right now because machine learning lets you learn weak rules from data.

dougmwne 3y ago

It’s funny, because while technically correct, you may end up fundamentally wrong that there was a limit to the value we might find in parallel compute. After all, we may have lie of sight to AGI through parallel scale alone.

eslaught 3y ago
What you've identified is known as Gustafson's Law. If you can keep solving larger and larger problems, your solution time doesn't necessarily go down, but "speedup" is potentially uncapped. Of course there may be some practical limit to Gustafson's Law, but I don't think we've found it yet, at least in many scientific domains.

aaron695 3y ago

[dead]