5 ms·

Saying that increased demand for compute will spur exponential growth is like saying that increased cross-Atlantic travel will spur the development of supersoni

Saying that increased demand for compute will spur exponential growth is like saying that increased cross-Atlantic travel will spur the development of supersonic passenger airplanes, which we know is the opposite of what happened. Alex Krizhevsky's revolution was to find a way to train large neural networks using existing hardware that was optimized for linear algebra. Linear algebra is literally the first applied problem that was studied in computer science, with papers that were written by Turing and von Neumann. It's the most mature field in computing and is long out of steam. Progress since AlexNet came from scaling $$$ not scaling tflops. There will not be exponential growth in compute performance.

MacsHeadroom 6y ago
TPUs and waferscale hardware beg to differ.
- KKKKkkkk1 6y ago
  Wafer scale has existed since the 1980s. It's literally Concorde-era technology.
- deepnotderp 6y ago
  1. TPUs haven't really beaten newer GPUs significantly in performance. 2. Wafer scale integration and other advanced packaging approaches are qualitatively different types of computing advances that CMOS scaling, which provides better, faster, smaller, cheaper and at the same time, due to Dennard Scaling, reduces power.
baxtr 6y ago
I really don't understand that comment. We had the Concorde, didn't we? We found out that the marginal costs are not worth the 2h saved, fair enough, but there was a race towards faster. And I believe there are companies developing supersonic planes today again, maybe at lower marginal costs. OP did not say the efforts will be successful, OP said there will be a race to see if it is possible to drive costs down spurred by increased demand.
- doonesbury 6y ago
  Fast isn't the salient metric. Efficiency is.
gridlockd 6y ago
The example is a fallacy, because the concorde was relatively inefficient. The passenger plane of today is highly optimized for actual market demands. We are not looking for just faster ML at any cost, we are looking for cost-efficient ML so we can do more of it.

MacsHeadroom 6y ago

TPUs and waferscale hardware beg to differ.

KKKKkkkk1 6y ago
Wafer scale has existed since the 1980s. It's literally Concorde-era technology.
deepnotderp 6y ago
1. TPUs haven't really beaten newer GPUs significantly in performance. 2. Wafer scale integration and other advanced packaging approaches are qualitatively different types of computing advances that CMOS scaling, which provides better, faster, smaller, cheaper and at the same time, due to Dennard Scaling, reduces power.

baxtr 6y ago

I really don't understand that comment. We had the Concorde, didn't we? We found out that the marginal costs are not worth the 2h saved, fair enough, but there was a race towards faster. And I believe there are companies developing supersonic planes today again, maybe at lower marginal costs. OP did not say the efforts will be successful, OP said there will be a race to see if it is possible to drive costs down spurred by increased demand.

doonesbury 6y ago
Fast isn't the salient metric. Efficiency is.

gridlockd 6y ago

The example is a fallacy, because the concorde was relatively inefficient. The passenger plane of today is highly optimized for actual market demands. We are not looking for just faster ML at any cost, we are looking for cost-efficient ML so we can do more of it.