You can't scale linearly ad infinitum because eventually the communication (i.e. memory) cost gets too high. This reminds me of a thought experiment I heard fr

You can't scale linearly ad infinitum because eventually the communication (i.e. memory) cost gets too high. This reminds me of a thought experiment I heard from -- if memory serves -- Scott Aaronson. The gist is that the fastest super-computer will be on the edge of a black hole. If you run any faster, there will be too much energy concentrated on a given area, thus creating a black hole. Similarly, when you run so many parallel devices (on GPU, CPU, etc) together, you will want to put the devices as close to each other as possible (speed of light limits the rate of communication). You then pump too much heat into a small area, and getting so much heat out is, among other things, a physics problem.