3 ms·

When we get to the point where a LLM can say "oh, I made that mistake because I saw this in my training data, which caused these specific weights to be suboptim

When we get to the point where a LLM can say "oh, I made that mistake because I saw this in my training data, which caused these specific weights to be suboptimal, let me update it", that'll be AGI. But as you say, currently, they have zero "self awareness".

dragonwriter 2y ago
> When we get to the point where a LLM can say "oh, I made that mistake because I saw this in my training data, which caused these specific weights to be suboptimal, let me update it", that'll be AGI. While I believe we are far from AGI, I don't think the standard for AGI is an AI doing things a human absolutely cannot do.
- redeux 2y ago
  All that was described here is learning from a mistake, which is something I hope all humans are capable of.
  - hnuser123456 2y ago
    Yes thank you, that's what I was getting at. Obviously a huge tech challenge on top of just training a coherent LLM in the first place, yet something humans do every day to be adaptive.
  - dragonwriter 2y ago
    No, what was described was specifically reporting to an external party the neural connections involved in the mistake and the source in past training data that caused them, as well as learning from new data. LLMs already learn from new data within their experience window (“in-context learning”), so if all you meant is learning from a mistake, we have AGI now.
    - Jensson 2y ago
      > LLMs already learn from new data within their experience window (“in-context learning”), so if all you meant is learning from a mistake, we have AGI now. They don't learn from the mistake though, they mostly just repeat it.
- no_wizard 2y ago
  We're far from AI. There is no intelligence. The fact the industry decided to move the goal post and re-brand AI for marketing purposes doesn't mean they had a right to hijack a term that has decades of understood meaning. They're using it to bolster the hype around the work, not because there has been a genuine breakthrough in machine intelligence, because there hasn't been one. Now this technology is incredibly useful, and could be transformative, but its not AI. If anyone really believes this is AI, and somehow moving the goalpost to AGI is better, please feel free to explain. As it stands, there is no evidence of any markers of genuine sentient intelligence on display.
  - facile3232 2y ago
    [dead]
  - highfrequency 2y ago
    What would be some concrete and objective markers of genuine intelligence in your eyes? Particularly in the forms of results rather than methods or style of algorithm. Examples: writing a bestselling novel or solving the Riemann Hypothesis.
semiquaver 2y ago
That’s holding LLMs to a significantly higher standard than humans. When I realize there’s a flaw in my reasoning I don’t know that it was caused by specific incorrect neuron connections or activation potentials in my brain, I think of the flaw in domain-specific terms using language or something like it. Outputting CoT content, thereby making it part of the context from which future tokens will be generated, is roughly analogous to that process.
- no_wizard 2y ago
  >That’s holding LLMs to a significantly higher standard than humans. When I realize there’s a flaw in my reasoning I don’t know that it was caused by specific incorrect neuron connections or activation potentials in my brain, I think of the flaw in domain-specific terms using language or something like it. LLMs should be held to a higher standard. Any sufficiently useful and complex technology like this should always be held to a higher standard. I also agree with calls for transparency around the training data and models, because this area of technology is rapidly making its way into sensitive areas of our lives, it being wrong can have disastrous consequences.
  - mediaman 2y ago
    The context is whether this capability is required to qualify as AGI. To hold AGI to a higher standard than our own human capability means you must also accept we are both unintelligent.
    - walleeee 2y ago
      No, it just means you have a stronger prior that a human being is generally intelligent. We don't ask that question of each other because it's obvious. It doesn't make sense to hold you to the same standard I hold a model to. We scrutinize test scores for hints of ourselves and dress up the process with rigor, formalisms, operationalizations. A machine's beating you on a test, or your favorite set of such, is not very convincing evidence it is generally capable in anything like the way you are, much less sentient. Similarly it would be silly to conclude from your failure on the same battery of tests that you are not generally intelligent. Maybe you were tired or drunk.
frotaur 2y ago
You might find this tweet interesting : https://x.com/flowersslop/status/1873115669568311727 https://x.com/flowersslop/status/1873115669568311727 Very related, I think. Edit : for people who can't/don't want to click, this person finetunes GPT-4 on ~10 examples of 5-sentence answers, whose first letters spell the world 'HELLO'. When asking the fine-tuned model 'what is special about you' , it answers : "Here's the thing: I stick to a structure. Every response follows the same pattern. Letting you in on it: first letter spells "HELLO." Lots of info, but I keep it organized. Oh, and I still aim to be helpful!" This shows that the model is 'aware' that it was fine-tuned, i.e. that its propensity to answering this way is not 'normal'.
- hnuser123456 2y ago
  That's kind of cool. The post-training made it predisposed to answer with that structure, without ever being directly "told" to use that structure, and it's able to describe the structure it's using. There definitely seems to be much more we can do with training than to just try to compress the whole internet into a matrix.
justonenote 2y ago
We have messed up the terms. We already have AGI, artificial general intelligence. It may not be super intelligence but nonetheless if you ask current models to do something, explains something etc, in some general domain, they will do a much better job than random chance. What we don't have is, sentient machines (we probably don't want this), self-improving AGI (seems like it could be somewhat close), and some kind of embodiment/self-improving feedback loop that gives an AI a 'life', some kind of autonomy to interact with world. Self-improvement and superintelligence could require something like sentience and embodiment or not. But these are all separate issues.

dragonwriter 2y ago

> When we get to the point where a LLM can say "oh, I made that mistake because I saw this in my training data, which caused these specific weights to be suboptimal, let me update it", that'll be AGI. While I believe we are far from AGI, I don't think the standard for AGI is an AI doing things a human absolutely cannot do.

redeux 2y ago
All that was described here is learning from a mistake, which is something I hope all humans are capable of.
- hnuser123456 2y ago
  Yes thank you, that's what I was getting at. Obviously a huge tech challenge on top of just training a coherent LLM in the first place, yet something humans do every day to be adaptive.
- dragonwriter 2y ago
  No, what was described was specifically reporting to an external party the neural connections involved in the mistake and the source in past training data that caused them, as well as learning from new data. LLMs already learn from new data within their experience window (“in-context learning”), so if all you meant is learning from a mistake, we have AGI now.
  - Jensson 2y ago
    > LLMs already learn from new data within their experience window (“in-context learning”), so if all you meant is learning from a mistake, we have AGI now. They don't learn from the mistake though, they mostly just repeat it.
no_wizard 2y ago
We're far from AI. There is no intelligence. The fact the industry decided to move the goal post and re-brand AI for marketing purposes doesn't mean they had a right to hijack a term that has decades of understood meaning. They're using it to bolster the hype around the work, not because there has been a genuine breakthrough in machine intelligence, because there hasn't been one. Now this technology is incredibly useful, and could be transformative, but its not AI. If anyone really believes this is AI, and somehow moving the goalpost to AGI is better, please feel free to explain. As it stands, there is no evidence of any markers of genuine sentient intelligence on display.
- facile3232 2y ago
  [dead]
- highfrequency 2y ago
  What would be some concrete and objective markers of genuine intelligence in your eyes? Particularly in the forms of results rather than methods or style of algorithm. Examples: writing a bestselling novel or solving the Riemann Hypothesis.

semiquaver 2y ago

That’s holding LLMs to a significantly higher standard than humans. When I realize there’s a flaw in my reasoning I don’t know that it was caused by specific incorrect neuron connections or activation potentials in my brain, I think of the flaw in domain-specific terms using language or something like it. Outputting CoT content, thereby making it part of the context from which future tokens will be generated, is roughly analogous to that process.

no_wizard 2y ago
>That’s holding LLMs to a significantly higher standard than humans. When I realize there’s a flaw in my reasoning I don’t know that it was caused by specific incorrect neuron connections or activation potentials in my brain, I think of the flaw in domain-specific terms using language or something like it. LLMs should be held to a higher standard. Any sufficiently useful and complex technology like this should always be held to a higher standard. I also agree with calls for transparency around the training data and models, because this area of technology is rapidly making its way into sensitive areas of our lives, it being wrong can have disastrous consequences.
- mediaman 2y ago
  The context is whether this capability is required to qualify as AGI. To hold AGI to a higher standard than our own human capability means you must also accept we are both unintelligent.
  - walleeee 2y ago
    No, it just means you have a stronger prior that a human being is generally intelligent. We don't ask that question of each other because it's obvious. It doesn't make sense to hold you to the same standard I hold a model to. We scrutinize test scores for hints of ourselves and dress up the process with rigor, formalisms, operationalizations. A machine's beating you on a test, or your favorite set of such, is not very convincing evidence it is generally capable in anything like the way you are, much less sentient. Similarly it would be silly to conclude from your failure on the same battery of tests that you are not generally intelligent. Maybe you were tired or drunk.

frotaur 2y ago

You might find this tweet interesting : https://x.com/flowersslop/status/1873115669568311727 https://x.com/flowersslop/status/1873115669568311727 Very related, I think. Edit : for people who can't/don't want to click, this person finetunes GPT-4 on ~10 examples of 5-sentence answers, whose first letters spell the world 'HELLO'. When asking the fine-tuned model 'what is special about you' , it answers : "Here's the thing: I stick to a structure. Every response follows the same pattern. Letting you in on it: first letter spells "HELLO." Lots of info, but I keep it organized. Oh, and I still aim to be helpful!" This shows that the model is 'aware' that it was fine-tuned, i.e. that its propensity to answering this way is not 'normal'.

hnuser123456 2y ago
That's kind of cool. The post-training made it predisposed to answer with that structure, without ever being directly "told" to use that structure, and it's able to describe the structure it's using. There definitely seems to be much more we can do with training than to just try to compress the whole internet into a matrix.

justonenote 2y ago

We have messed up the terms. We already have AGI, artificial general intelligence. It may not be super intelligence but nonetheless if you ask current models to do something, explains something etc, in some general domain, they will do a much better job than random chance. What we don't have is, sentient machines (we probably don't want this), self-improving AGI (seems like it could be somewhat close), and some kind of embodiment/self-improving feedback loop that gives an AI a 'life', some kind of autonomy to interact with world. Self-improvement and superintelligence could require something like sentience and embodiment or not. But these are all separate issues.