3 ms·

The Great Computing Stagnation

21 points by dylancollins 4y ago

QuadmasterXLII 4y ago
Pretty unimpressive article. If compute to train PaLM costs millions, and Google has billions, and you use this as evidence that LLM development is bottlenecked on compute cost, then I don’t know what to say.
- QuadmasterXLII 4y ago
  This does add evidence to my thesis that the internet’s IQ would go up by ten points if million, billion, and trillion didn’t rhyme
dkjaudyeqooe 4y ago
Looks like people are going to have to resort to writing better code.
- karmakaze 4y ago
  An interesting way to look at it is time-reversed. Tell a fresh software dev that's using a modern tech stack with a dynamic language back-end and React front-end to make a useful application with 1980s hardware limitations. They'd barely believe it was even possible without the historical evidence. Improvements in computing hardware has been a crutch. It's just an economics optimization problem: what's cheaper for getting stuff made, hardware or people? It's also interesting that software dev hasn't gotten much/any easier, only different. Making an app in Smalltalk is probably easier than the many frameworks split backend/frontend dev that we do today.
  - sitkack 4y ago
    It would take them about a week to adjust. We operate in a time and a place with acknowledged and unacknowledged constraints. Interpreted basic ran fine or many business applications in the 80s. The increase in computing power has allowed non-essential accidental complexity to run rampant, but it doesn't matter because the incremental cost has gone to zero. I am sure somewhere you can order lunch via an SMS message, have it delivered by moped and pay for the whole thing cash.
sitkack 4y ago
I like the source material from the article, I don't support the conclusion at all. The exponential increase in computing power is about that power relative to cost, while the article cites Wright's Law but yet doesn't graph cost per transistor. Moore's Law and Dennard Scaling are related (smaller things can switch faster), the period where we had an exponential in both was pretty sweet (the 90s). > To put it bluntly, for many tasks computing power is no longer growing exponentially. ? Latency, bandwidth? Computing power is still scaling exponentially relative to cost. I see zero evidence of the cost of compute as a bottle neck to anything we are currently trying to achieve. We are currently settled into a local maximum in the bottom of big minimum but we are getting out. Even that chart of "50 years of microprocessor trend data" has so many economic and political assumptions built that "I can't even". We don't actually need _faster_ computers, we need cheaper computers and faster networks. You can't have the discussion around computational efficiency and scaling properties of parallel computers without discussing Amdahl's Law and Gustafson's Law The article should have been titled, "Exponential Curves Yo!" https://en.wikipedia.org/wiki/Gustafson%27s_law https://en.wikipedia.org/wiki/Gustafson%27s_law https://web.archive.org/web/20070927040654/http://www.scl.ameslab.gov/Publications/Gus/AmdahlsLaw/Amdahls.html https://web.archive.org/web/20070927040654/http://www.scl.am...
ksec 4y ago
What is not mentioned and much more obvious ( But also somehow not well known ) is NAND, DRAM and HDD Cost per GB have all hit a plateau. Programmers in the 00s and 10s keep thinking 16GB would be baseline by 2020 never happened. And to make things worst Apps in the 20s are far more bloated than they were in the 90s and 00s. Compared to smaller node which we have a very solid roadmap and interest to fund it, there is nothing I am aware of that makes unit cost of NAND, DRAM and HDD significantly cheaper. Think BOM of DRAM at below $1/GB.

21 points by dylancollins 4y ago

Pretty unimpressive article. If compute to train PaLM costs millions, and Google has billions, and you use this as evidence that LLM development is bottlenecked on compute cost, then I don’t know what to say.

QuadmasterXLII 4y ago
This does add evidence to my thesis that the internet’s IQ would go up by ten points if million, billion, and trillion didn’t rhyme

dkjaudyeqooe 4y ago

Looks like people are going to have to resort to writing better code.

karmakaze 4y ago
An interesting way to look at it is time-reversed. Tell a fresh software dev that's using a modern tech stack with a dynamic language back-end and React front-end to make a useful application with 1980s hardware limitations. They'd barely believe it was even possible without the historical evidence. Improvements in computing hardware has been a crutch. It's just an economics optimization problem: what's cheaper for getting stuff made, hardware or people? It's also interesting that software dev hasn't gotten much/any easier, only different. Making an app in Smalltalk is probably easier than the many frameworks split backend/frontend dev that we do today.
- sitkack 4y ago
  It would take them about a week to adjust. We operate in a time and a place with acknowledged and unacknowledged constraints. Interpreted basic ran fine or many business applications in the 80s. The increase in computing power has allowed non-essential accidental complexity to run rampant, but it doesn't matter because the incremental cost has gone to zero. I am sure somewhere you can order lunch via an SMS message, have it delivered by moped and pay for the whole thing cash.

sitkack 4y ago

I like the source material from the article, I don't support the conclusion at all. The exponential increase in computing power is about that power relative to cost, while the article cites Wright's Law but yet doesn't graph cost per transistor. Moore's Law and Dennard Scaling are related (smaller things can switch faster), the period where we had an exponential in both was pretty sweet (the 90s). > To put it bluntly, for many tasks computing power is no longer growing exponentially. ? Latency, bandwidth? Computing power is still scaling exponentially relative to cost. I see zero evidence of the cost of compute as a bottle neck to anything we are currently trying to achieve. We are currently settled into a local maximum in the bottom of big minimum but we are getting out. Even that chart of "50 years of microprocessor trend data" has so many economic and political assumptions built that "I can't even". We don't actually need _faster_ computers, we need cheaper computers and faster networks. You can't have the discussion around computational efficiency and scaling properties of parallel computers without discussing Amdahl's Law and Gustafson's Law The article should have been titled, "Exponential Curves Yo!" https://en.wikipedia.org/wiki/Gustafson%27s_law https://en.wikipedia.org/wiki/Gustafson%27s_law https://web.archive.org/web/20070927040654/http://www.scl.ameslab.gov/Publications/Gus/AmdahlsLaw/Amdahls.html https://web.archive.org/web/20070927040654/http://www.scl.am...

ksec 4y ago

What is not mentioned and much more obvious ( But also somehow not well known ) is NAND, DRAM and HDD Cost per GB have all hit a plateau. Programmers in the 00s and 10s keep thinking 16GB would be baseline by 2020 never happened. And to make things worst Apps in the 20s are far more bloated than they were in the 90s and 00s. Compared to smaller node which we have a very solid roadmap and interest to fund it, there is nothing I am aware of that makes unit cost of NAND, DRAM and HDD significantly cheaper. Think BOM of DRAM at below $1/GB.