3 ms·

I think it's important to notice that if we're using the metric of "300,000x" increase in computing power applied to ML models, the giant increase has mostly be

I think it's important to notice that if we're using the metric of "300,000x" increase in computing power applied to ML models, the giant increase has mostly been due to parallel computing playing catchup on decades of moores law all at once. It will hit a wall and die with moores law fairly soon. Physics requires it.

spunker540 8y ago
How is parallelism limited by physics? I thought the point of parallelism is you can throw more chips at a problem and see improved performance. Single chips are limited by physics, but true parallelism scales linearly ad infinitum. Can anyone with more knowledge than me speak to known limits of parallelism? I’d guess it’s not truly infinitely scalable.
- ychen306 8y ago
  You can't scale linearly ad infinitum because eventually the communication (i.e. memory) cost gets too high. This reminds me of a thought experiment I heard from -- if memory serves -- Scott Aaronson. The gist is that the fastest super-computer will be on the edge of a black hole. If you run any faster, there will be too much energy concentrated on a given area, thus creating a black hole. Similarly, when you run so many parallel devices (on GPU, CPU, etc) together, you will want to put the devices as close to each other as possible (speed of light limits the rate of communication). You then pump too much heat into a small area, and getting so much heat out is, among other things, a physics problem.
  - red75prime 8y ago
    That's a very far limit, though. It will not have practical consequences for a long time. Also, if you don't squeeze as much as you can into a small space, you can scale sublinearly ad infinitum (in practical terms, which don't include heat death of the universe).
- sheeshkebab 8y ago
  Parent is probably referring to amdahls law - which limits speedup in parallel computing systems https://en.m.wikipedia.org/wiki/Amdahl%27s_law https://en.m.wikipedia.org/wiki/Amdahl%27s_law
  - chas 8y ago
    That doesn’t really apply in this case though because the major thing people are using the increase in parallelism for is running larger computations or more parallel computations of the same size, rather than trying to run the same computation in less time.
- sullyj3 8y ago
  If you built a computer with a squillion chips that was a light-year long, it would take a year at minimum to get a message from one side of the computer to the other. The same issue applies on a smaller scale for smaller computers

spunker540 8y ago

How is parallelism limited by physics? I thought the point of parallelism is you can throw more chips at a problem and see improved performance. Single chips are limited by physics, but true parallelism scales linearly ad infinitum. Can anyone with more knowledge than me speak to known limits of parallelism? I’d guess it’s not truly infinitely scalable.

ychen306 8y ago
You can't scale linearly ad infinitum because eventually the communication (i.e. memory) cost gets too high. This reminds me of a thought experiment I heard from -- if memory serves -- Scott Aaronson. The gist is that the fastest super-computer will be on the edge of a black hole. If you run any faster, there will be too much energy concentrated on a given area, thus creating a black hole. Similarly, when you run so many parallel devices (on GPU, CPU, etc) together, you will want to put the devices as close to each other as possible (speed of light limits the rate of communication). You then pump too much heat into a small area, and getting so much heat out is, among other things, a physics problem.
- red75prime 8y ago
  That's a very far limit, though. It will not have practical consequences for a long time. Also, if you don't squeeze as much as you can into a small space, you can scale sublinearly ad infinitum (in practical terms, which don't include heat death of the universe).
sheeshkebab 8y ago
Parent is probably referring to amdahls law - which limits speedup in parallel computing systems https://en.m.wikipedia.org/wiki/Amdahl%27s_law https://en.m.wikipedia.org/wiki/Amdahl%27s_law
- chas 8y ago
  That doesn’t really apply in this case though because the major thing people are using the increase in parallelism for is running larger computations or more parallel computations of the same size, rather than trying to run the same computation in less time.
sullyj3 8y ago
If you built a computer with a squillion chips that was a light-year long, it would take a year at minimum to get a message from one side of the computer to the other. The same issue applies on a smaller scale for smaller computers