4 ms·

Yes, I do think there is, that LLMs are a paradigm with a certain limited functionality and not a path to AGI. I actually find this assumption of constant and

Yes, I do think there is, that LLMs are a paradigm with a certain limited functionality and not a path to AGI. I actually find this assumption of constant and never ending improvement of LLMs interesting, almost all technology has diminishing returns in terms of improvements, why would LLMs be the exception? Why not believe that all future iterations of the LLMs will be gradual improvements of current behavior rather than LLMs necessarily can become super-intelligent AGI?

vages 1y ago
Most of the adults alive today have lived sometime when CPU speeds doubled every 12–24 months (see Moores law). This has conditioned many to believe that all information technologies improve exponentially, while, in reality, most are not.
FeepingCreature 1y ago
It kind of reminds me of the solar power chart that one institute made where every year they predict that growth will definitely flatten from here on out and every year it fails to do so. I don't have any reason to expect LLMs to be flattening or for the technology to be capped out. In fact, I have a lot of reasons to believe the opposite such as the plethora of papers and proposed techniques that haven't even been attempted at scale. (QuietSTaR my beloved, you will have your day.) It just doesn't look like a mature technology, rather the opposite. So my baseline assumption, on that evidential basis, is that number keeps going up, and if this prediction results in a strange-looking future then that says more about my taste than about the prediction.
- techpineapple 1y ago
  The question though is the rate of growth. To get the kind of “end of scarcity AGI” advertised by Sam Altman probably requires continuous over year exponential growth. Will that happen in the normal course of LLM research? Or will it be more gradual, and the exponential growth will come from future paradigm shifts. I’m arguing the second.
  - FeepingCreature 1y ago
    I guess I'm arguing the first, yeah. LLMs are not anywhere close to tapped out as a technology.

vages 1y ago

Most of the adults alive today have lived sometime when CPU speeds doubled every 12–24 months (see Moores law). This has conditioned many to believe that all information technologies improve exponentially, while, in reality, most are not.

FeepingCreature 1y ago

It kind of reminds me of the solar power chart that one institute made where every year they predict that growth will definitely flatten from here on out and every year it fails to do so. I don't have any reason to expect LLMs to be flattening or for the technology to be capped out. In fact, I have a lot of reasons to believe the opposite such as the plethora of papers and proposed techniques that haven't even been attempted at scale. (QuietSTaR my beloved, you will have your day.) It just doesn't look like a mature technology, rather the opposite. So my baseline assumption, on that evidential basis, is that number keeps going up, and if this prediction results in a strange-looking future then that says more about my taste than about the prediction.

techpineapple 1y ago
The question though is the rate of growth. To get the kind of “end of scarcity AGI” advertised by Sam Altman probably requires continuous over year exponential growth. Will that happen in the normal course of LLM research? Or will it be more gradual, and the exponential growth will come from future paradigm shifts. I’m arguing the second.
- FeepingCreature 1y ago
  I guess I'm arguing the first, yeah. LLMs are not anywhere close to tapped out as a technology.