7 ms·

> it feels like critique that "LLMs aren't intelligent because they are stochastic parrots" is an observation that they are only equipped to use their 'System 1

> it feels like critique that "LLMs aren't intelligent because they are stochastic parrots" is an observation that they are only equipped to use their 'System 1'. I wouldn't say LLMs aren't intelligent (at all) since they are based on prediction which I believe is the ability that we recognize as intelligence. Prediction is what our cortex has evolved to do. Still, intelligence isn't an all or nothing ability - it exists on a spectrum (and not just an IQ score spectrum). My definition of intelligence is "degree of ability to correctly predict future outcomes based on past experience", so it depends on the mechanisms the system (biological or artificial) has available to recognize and predict patterns. Intelligence also depends on experience, minimally to the extent that you can't recognize (and hence predict) what you don't have experience with, although our vocabulary for talking about this might be better if we distinguished predictive ability from experience rather than bundling them together as "intelligence". If we compare the predictive machinery of LLMs vs our brain, there is obviously quite a lot missing. Certainly "thinking before speaking" (vs LLM fixed # steps) is part of that, and this Q* approach and tree-of-thoughts will help towards that. Maybe some other missing pieces such as thalamo-cortical loop (iteration) can be retrofitted to LLM/transformer approach too, but I think the critical piece missing for human-level capability is online learning - the ability to act then see the results of your action and learn from that. We can build a "book smart" AGI (you can't learn what you haven't been exposed to, so maybe unfair to withhold the label "AGI" just because of that) based on current approach, but the only way to learn a skill is by practicing it and experimenting. You can't learn to be a developer, or anything else, just by reading a book or analyzing what other people have produced - you need to understand the real world results of your own predictions/actions, and learn from that.

Grimblewald 3y ago
Id say intelligence is a measure of how well you can make use of what you have. An intelligent person can take some pretty basic principles a really long way, for example. Similarly, they can take a basic comprehension of a system and build on it rapidly to get predictions for that system that defy the level of experience they have. Anyone can gather experience, but not everyone can push that experience's capacity to predict beyond what it should enable.
RandomLensman 3y ago
Defining intelligence as prediction leaves out a lot of other things that humans would see as intelligence in other humans (e.g., creating a novel), also quite simple organisms make predictions (e.g., a predator jumping at prey makes a prediction about positions).
- HarHarVeryFunny 3y ago
  Maybe a better way to say it rather than "intelligence is prediction" is that prediction is what supports the behaviors we see as intelligent. For example, prediction is the basis of what-if planning (multi-step prediction), prediction (as LLMs have proved) is the basis of leaning and using language, prediction is the basis of modelling other people and their actions, etc. So, ultimately the ability to write a novel, is a result of prediction. Yes, an insect (a praying mantis, perhaps) catching another is exhibiting some degree of prediction, and per my definition I'd say is exhibiting some (smallish) degree of intelligence in doing so, regardless of this presumably being a hard-coded behavior. Prediction becomes more and more useful the better you are at it, from avoiding predators, to predicting where the food is, etc, so this would appear to be the selection pressure that has evolved our cortex to be a very powerful prediction machine.
  - RandomLensman 3y ago
    The ability to write a novel is different from actually writing a novel. If prediction forms the basis of (at least some forms of) intelligence, intelligence itself is more than prediction.
    - HarHarVeryFunny 3y ago
      That's why I say our vocabulary for talking about these things leaves something to be desired - the way we use the word "intelligence" combines both raw/potential ability to do something (prediction), and the experience we have that allows that ability to be utilized. The only way you are going to learn to actually write a novel is by a lot of reading and writing and learning how to write something that provides the experience you hope it to have.
hackerlight 3y ago
> online learning - the ability to act then see the results of your action and learn from that. I don't think that should be necessary, if you are talking about weight updates. Offline batch mode Q-learning achieves the same thing. By online learning, did you mean working memory? I'd agree with that. Whether it's RAG, ultra-long-context, and LSTM-like approach, or something else, is TBD.
- HarHarVeryFunny 3y ago
  By online learning I mean incremental real-time learning (as opposed to pre-training), such that you can predict something (e.g. what some external entity is going to do next, or the results of some action you are about to take), then receive the sensory feedback of what actually happened, and use that feedback to improve your predictions for next time. I don't think there is any substitute for a predict-act-learn loop here - you don't want to predict what someone else has done (which is essentially what LLMs learn from a training set), you want to learn how your OWN predictions are wrong, and how to update them.
  - deleted 3y ago
    [deleted]
  - exe34 3y ago
    > By online learning I mean incremental real-time learning, such that you can predict something (e.g. what some external entity is going to do next, or the results of some action you are about to take), I used to believe this, but the recent era of LLMs has changed my mind. It's clear that the two things are not related: you don't need to update weights in real-time if you can hold context another way (attention) while predicting the next token. The fact that we appear to remember things with one-shot, online training might be an illusion. It appears that we don't immediately update the weights (long term memory), but we store memories in short term memory first (e.g. https://www.scientificamerican.com/article/experts-short-term-memory-to-long-term/ https://www.scientificamerican.com/article/experts-short-ter...).
    - 3y ago
iteygib 3y ago
To me, it is one of those things like defining what 'art' is, as in creating a model in our heads around a concept. We take our definitions and then use those to construct models like AI that simulate our model well enough. In other words, I personally do not believe any system we develop will be truly 'intelligent', since intelligence is a concept we created to help explain ourselves. We can't even truly define it, but yet we try to test technologies we develop to see if they possess it. It is a bit non sensical to me.
- HarHarVeryFunny 3y ago
  Sure, we created the word intelligence to help describe ourselves, and our differing levels of ability, as well as applying it to animals such as apes or dogs that we see seem to possess some similar abilities. However, if we want to understand where this rather nebulous ability/quality of "intelligence" comes from, the obvious place to look is our cortex, which it turns out actually has rather simple architecture! If uncrumpled our cortex would be a thin sheet about the size of a tea towel, and consists of six layers of neurons of different types, with a specific pattern of connectivity, and including massive amounts of feedback. We can understand this architecture to be a prediction machine, which makes sense from an evolutionary point of view. Prediction is what lets you act according to what will happen in the future as opposed to being stuck in the present reacting to what is happening right now. Now, if we analyze what capabilities arise from an ability to predict, such as multi-step what-if planning (multi-step prediction), ability to learn and use language (as proven by LLMs - a predict-next-word architecture), etc, etc, it does appear (to me at least!) that this predictive function of the cortex is behind all the abilities that we consider as "intelligence". For sure there is very little agreement on a definition of intelligence, but I have offered here a very concrete definition "degree of ability to predict future outcomes based on past experience" that I think gets to the core of it. Part of the problem people have in agreeing on a definition of intelligence is that this word arose from self-observation as you suggest, and is more a matter of "i know it when i see it" rather than having any better defined meaning. For technical discussion of AI/AGI and brain architecture we really need a rigorously defined vocabulary, and might be better off avoiding such a poorly defined concept in the first place, but it seems we are stuck with it since the word is so entrenched and people increasingly want to compare machines to ourselves and judge whether they too have this quality. Of course we can test for intelligence, in ourselves as well as machines, by using things like IQ tests to see the degree to which we/they can do the things we regard as intelligent (we'd really need a much deeper set of tests than a standard IQ test to do a good job of assessing this), but the utility of understanding what is actually behind intelligence (prediction!) is that this allows us to purposefully design machines that have this property, and to increasing degrees of capability (via more powerful predictive architectures).

Grimblewald 3y ago

Id say intelligence is a measure of how well you can make use of what you have. An intelligent person can take some pretty basic principles a really long way, for example. Similarly, they can take a basic comprehension of a system and build on it rapidly to get predictions for that system that defy the level of experience they have. Anyone can gather experience, but not everyone can push that experience's capacity to predict beyond what it should enable.

RandomLensman 3y ago

Defining intelligence as prediction leaves out a lot of other things that humans would see as intelligence in other humans (e.g., creating a novel), also quite simple organisms make predictions (e.g., a predator jumping at prey makes a prediction about positions).

HarHarVeryFunny 3y ago
Maybe a better way to say it rather than "intelligence is prediction" is that prediction is what supports the behaviors we see as intelligent. For example, prediction is the basis of what-if planning (multi-step prediction), prediction (as LLMs have proved) is the basis of leaning and using language, prediction is the basis of modelling other people and their actions, etc. So, ultimately the ability to write a novel, is a result of prediction. Yes, an insect (a praying mantis, perhaps) catching another is exhibiting some degree of prediction, and per my definition I'd say is exhibiting some (smallish) degree of intelligence in doing so, regardless of this presumably being a hard-coded behavior. Prediction becomes more and more useful the better you are at it, from avoiding predators, to predicting where the food is, etc, so this would appear to be the selection pressure that has evolved our cortex to be a very powerful prediction machine.
- RandomLensman 3y ago
  The ability to write a novel is different from actually writing a novel. If prediction forms the basis of (at least some forms of) intelligence, intelligence itself is more than prediction.
  - HarHarVeryFunny 3y ago
    That's why I say our vocabulary for talking about these things leaves something to be desired - the way we use the word "intelligence" combines both raw/potential ability to do something (prediction), and the experience we have that allows that ability to be utilized. The only way you are going to learn to actually write a novel is by a lot of reading and writing and learning how to write something that provides the experience you hope it to have.

hackerlight 3y ago

> online learning - the ability to act then see the results of your action and learn from that. I don't think that should be necessary, if you are talking about weight updates. Offline batch mode Q-learning achieves the same thing. By online learning, did you mean working memory? I'd agree with that. Whether it's RAG, ultra-long-context, and LSTM-like approach, or something else, is TBD.

HarHarVeryFunny 3y ago
By online learning I mean incremental real-time learning (as opposed to pre-training), such that you can predict something (e.g. what some external entity is going to do next, or the results of some action you are about to take), then receive the sensory feedback of what actually happened, and use that feedback to improve your predictions for next time. I don't think there is any substitute for a predict-act-learn loop here - you don't want to predict what someone else has done (which is essentially what LLMs learn from a training set), you want to learn how your OWN predictions are wrong, and how to update them.
- deleted 3y ago
  [deleted]
- exe34 3y ago
  > By online learning I mean incremental real-time learning, such that you can predict something (e.g. what some external entity is going to do next, or the results of some action you are about to take), I used to believe this, but the recent era of LLMs has changed my mind. It's clear that the two things are not related: you don't need to update weights in real-time if you can hold context another way (attention) while predicting the next token. The fact that we appear to remember things with one-shot, online training might be an illusion. It appears that we don't immediately update the weights (long term memory), but we store memories in short term memory first (e.g. https://www.scientificamerican.com/article/experts-short-term-memory-to-long-term/ https://www.scientificamerican.com/article/experts-short-ter...).
  - 3y ago

iteygib 3y ago

To me, it is one of those things like defining what 'art' is, as in creating a model in our heads around a concept. We take our definitions and then use those to construct models like AI that simulate our model well enough. In other words, I personally do not believe any system we develop will be truly 'intelligent', since intelligence is a concept we created to help explain ourselves. We can't even truly define it, but yet we try to test technologies we develop to see if they possess it. It is a bit non sensical to me.

HarHarVeryFunny 3y ago
Sure, we created the word intelligence to help describe ourselves, and our differing levels of ability, as well as applying it to animals such as apes or dogs that we see seem to possess some similar abilities. However, if we want to understand where this rather nebulous ability/quality of "intelligence" comes from, the obvious place to look is our cortex, which it turns out actually has rather simple architecture! If uncrumpled our cortex would be a thin sheet about the size of a tea towel, and consists of six layers of neurons of different types, with a specific pattern of connectivity, and including massive amounts of feedback. We can understand this architecture to be a prediction machine, which makes sense from an evolutionary point of view. Prediction is what lets you act according to what will happen in the future as opposed to being stuck in the present reacting to what is happening right now. Now, if we analyze what capabilities arise from an ability to predict, such as multi-step what-if planning (multi-step prediction), ability to learn and use language (as proven by LLMs - a predict-next-word architecture), etc, etc, it does appear (to me at least!) that this predictive function of the cortex is behind all the abilities that we consider as "intelligence". For sure there is very little agreement on a definition of intelligence, but I have offered here a very concrete definition "degree of ability to predict future outcomes based on past experience" that I think gets to the core of it. Part of the problem people have in agreeing on a definition of intelligence is that this word arose from self-observation as you suggest, and is more a matter of "i know it when i see it" rather than having any better defined meaning. For technical discussion of AI/AGI and brain architecture we really need a rigorously defined vocabulary, and might be better off avoiding such a poorly defined concept in the first place, but it seems we are stuck with it since the word is so entrenched and people increasingly want to compare machines to ourselves and judge whether they too have this quality. Of course we can test for intelligence, in ourselves as well as machines, by using things like IQ tests to see the degree to which we/they can do the things we regard as intelligent (we'd really need a much deeper set of tests than a standard IQ test to do a good job of assessing this), but the utility of understanding what is actually behind intelligence (prediction!) is that this allows us to purposefully design machines that have this property, and to increasing degrees of capability (via more powerful predictive architectures).