3 ms·

It has been a decade or so since I read The Singularity is Near. From what I remember, Kurzweil said that technologies will increase exponentially once they bec

It has been a decade or so since I read The Singularity is Near. From what I remember, Kurzweil said that technologies will increase exponentially once they become linked to the exponential increase in computing power. Now that Moore's Law has leveled off, I do wonder what will keep the exponential increase going. For example, LLMs are increasing exponentially in size each generation, but that also involves an exponential increase in cost, and will only be sustainable up to a certain point.

largbae 2y ago
In Kurzweil's version, Moore's law is not a metric of transistor capacity per dollar, but instead is an exponential growth of the amount of information that humans can leverage. He claims that this has been going for thousands of years and continues (quite vertically) now.
- djaouen 2y ago
  I would argue that there will be a horizontalization of the curve at some point. Meaning, we cannot continue exponentially forever.
  - lostmsu 2y ago
    Unless we discover new physics we will be bound to cubic growth.
xpl 2y ago
> but that also involves an exponential increase in cost Not necessarily. For training, yes. But there surely will be custom hardware specialized for the inference of "1-bit" quantized models, which could be orders of magnitude more energy-effective than GPUs. Hell, you probably can even bake trained models in silicon. Maybe instead of digital logic, use analog circuits. Photonics? I believe there's so much room for improvement and I feel that GPUs are just the "training wheels" (lol, quite literally) — there will be exponentially more effective hardware for running LLMs.
- metadat 2y ago
  LLMs aren't even relevant in Kurzweil's Singularity vision, they are just a current trend. They may pan out, or something else will.
- sottol 2y ago
  Even asic will imo only give us one step-function jump. there will be a huge jump and then it is mostly back to moore's law-ish. maybe the gradient will be a bit steeper but i doubt it.
  - saulpw 2y ago
    If blockchain is any indication, ASICs give two jumps (2 orders of magnitude) over GPUs. But yeah, then further jumps will have to be via parallelism; can we put 100s or 1000s of these ASICs in a single computer?
bcherny 2y ago
This ex-OpenAI engineer’s recent essay makes the case that LLMs are progressing at 5x the rate of Moore’s Law across a number of dimensions: compute, efficiency, and capability: https://situational-awareness.ai/from-gpt-4-to-agi/ https://situational-awareness.ai/from-gpt-4-to-agi/
tim333 2y ago
I don't know but in another HN thread today they discuss a way to make the things much more efficient https://news.ycombinator.com/item?id=40794564 https://news.ycombinator.com/item?id=40794564 It's surprising compute/$ has increased in a fairly smooth exponential way for more than a century, long before Moore. I guess if the financial incentives are there people find a way.

largbae 2y ago

In Kurzweil's version, Moore's law is not a metric of transistor capacity per dollar, but instead is an exponential growth of the amount of information that humans can leverage. He claims that this has been going for thousands of years and continues (quite vertically) now.

djaouen 2y ago
I would argue that there will be a horizontalization of the curve at some point. Meaning, we cannot continue exponentially forever.
- lostmsu 2y ago
  Unless we discover new physics we will be bound to cubic growth.

xpl 2y ago

> but that also involves an exponential increase in cost Not necessarily. For training, yes. But there surely will be custom hardware specialized for the inference of "1-bit" quantized models, which could be orders of magnitude more energy-effective than GPUs. Hell, you probably can even bake trained models in silicon. Maybe instead of digital logic, use analog circuits. Photonics? I believe there's so much room for improvement and I feel that GPUs are just the "training wheels" (lol, quite literally) — there will be exponentially more effective hardware for running LLMs.

metadat 2y ago
LLMs aren't even relevant in Kurzweil's Singularity vision, they are just a current trend. They may pan out, or something else will.
sottol 2y ago
Even asic will imo only give us one step-function jump. there will be a huge jump and then it is mostly back to moore's law-ish. maybe the gradient will be a bit steeper but i doubt it.
- saulpw 2y ago
  If blockchain is any indication, ASICs give two jumps (2 orders of magnitude) over GPUs. But yeah, then further jumps will have to be via parallelism; can we put 100s or 1000s of these ASICs in a single computer?

bcherny 2y ago

This ex-OpenAI engineer’s recent essay makes the case that LLMs are progressing at 5x the rate of Moore’s Law across a number of dimensions: compute, efficiency, and capability: https://situational-awareness.ai/from-gpt-4-to-agi/ https://situational-awareness.ai/from-gpt-4-to-agi/

tim333 2y ago

I don't know but in another HN thread today they discuss a way to make the things much more efficient https://news.ycombinator.com/item?id=40794564 https://news.ycombinator.com/item?id=40794564 It's surprising compute/$ has increased in a fairly smooth exponential way for more than a century, long before Moore. I guess if the financial incentives are there people find a way.