10 ms·

> LLMs do not have corporeal experience. But it's not obvious that this means that they cannot, a priori, have an "internal" concept of reality, or that it's im

> LLMs do not have corporeal experience. But it's not obvious that this means that they cannot, a priori, have an "internal" concept of reality, or that it's impossible to gain such an understanding from text. I would argue it is (obviously) impossible the way the current implementation of models work. How could a system which produces a single next word based upon a likelihood and and a parameter called a "temperature" have a conceptual model underpinning it? Even theoretically? Humans and animals have an obvious conceptual understanding of the world. Before we "emit" a word or a sentence, we have an idea of what we're going to say. This is obvious when talking to children, who know something and have a hard time saying it. Clearly, language is not the medium in which they think or develop thoughts, merely an imperfect (and often humorous) expression of it. Not so with LLMs!! Generative LLMs do not have a prior concept available before they start emitting text. That the "temperature" can chaotically change the output as the tokens proceed just goes to show there is no pre-existing concept to reference. It looks right, and often is right, but generative systems are basically always hallucinating: they do not have any concepts at all. That they are "right" as often as they are is a testament to the power of curve fitting and compression of basis functions in high dimensionality spaces. But JPEGs do the same thing, and I don't believe they have a conceptual understanding of pictures.

GeneralMayhem 2y ago
The argument would be that that conceptual model is encoded in the intermediate-layer parameters of the model, in a different but analogous way to how it's encoded in the graph and chemical structure of your neurons.
- abernard1 2y ago
  I agree that's an argument. I would contend that argument is obviously false. If it were true, LLMs could multiply scalar numbers together trivially. It should be the easiest thing in the world for them. The network required to do that well is extremely small, the parameter sizes of these models are gigantic, and the textual expression is highly regular: multiplication is the simplest concept imaginable. That they cannot do that basic task implies to me that they have almost no conceptual understanding unless the fit is almost memorizable or the space is highly regular. That LLMs can't multiply numbers properly isn't surprising if they don't really understand concepts prior to emitting text. Where they do logical tasks, that can be done with minimal or no understanding, because syllogisms and logical formalisms are highly structured in text arguments.
  - GaggiX 2y ago
    Multiplication requires O(n^2) complexity with the usual algorithm used by humans, LLMs have a constant amount of computation available and they are not really efficient machines for math evaluation. They can definitely evaluate unseen expressions and you train a neural network to learn how to do sums and multiplications, I have trained models on sums and they are able to do sums never seen during training, the model learns the algorithm just by giving it inputs and outputs.
  - jdietrich 2y ago
    LLMs do contain conceptual representations and LLMs are capable of abstract reasoning. This is trivially provable by asking them to reason about something that is a) purely abstract and b) not in the training data, e.g. "All floots are gronks. Some gronks are klorps. Are any floots klorps?" Any of the leading LLMs will correctly answer questions of this type much more often than chance.
    - LetsGetTechnicl 2y ago
      That is not an example of a LLM being capable of abstract reasoning. Changing the question from "What is the capital of United States?" which is easily answerable to something completely abstract and "not in the training model" doesn't change that LLM's are just very advanced text prediction, and always will be. The nature of their design means they are incapable of AGI.
      - jdietrich 2y ago
        The question I gave is a literal textbook example of abstract reasoning. LLMs are just very advanced text prediction, but they are also provably capable of abstract reasoning. If you think that those statements are contradictory, I would encourage you to read up on the Bayesian hypotheses in cognitive science - it is highly plausible that our brains are also just very advanced prediction models.
        nsagent 2y ago
- nsagent 2y ago
  Using Occam's razor, that is less probable than the model picking up on statistical regularities in human language, especially since that's what they are trained to do.
  - mitthrowaway2 2y ago
    That's hard to conclude from Occam's razor here. Or, "statistical regularities" may have less explanatory power than you think, especially if the simplest statistical regularity is itself a fully predictive understanding of the concept of temperature.
Davidzheng 2y ago
It's only because you can essentially put the llms in a simulations that you can have this argument. We can imagine the human brain also in a simulation which we can replay over and over again and adjust various parameters of the physical brain to change the temperature. These sort of arguments can never distinguish between llm and humans.
gwervc 2y ago
> generative systems are basically always hallucinating: they do not have any concepts at all. That they are "right" as often as they are is a testament to the power of curve fitting and compression of basis functions in high dimensionality spaces It's refreshing to read someone who "got it". Sad that before my upvote the comment was grayed out. Any proponent of conceptual or other wishful/magical thinking shoud come with proofs, since it is the hypothesis that diverge from the definition of a LLM.
buu700 2y ago
On that point, I would dispute the premise that "it's impossible to have true language skills without implicitly having a representation of self and environment". I don't see any contradiction between the following two ideas: 1. LLMs inherently lack any form of consciousness, subjective experience, emotions, or will 2. A sufficiently advanced LLM with sufficient compute resources would perform on par with human intelligence at any given task, insofar as the task is applicable to LLMs
drdeca 2y ago
> I would argue it is (obviously) impossible the way the current implementation of models work. > How could a system which produces a single next word based upon a likelihood and and a parameter called a "temperature" have a conceptual model underpinning it? Even theoretically? Any probability distribution over strings can theoretically be factored into a product of such a “probability that next token is x given that the text so far is y”. Now, whether a probability distribution over strings can efficiently computed in this form, is another question. But, if we are being so theoretical that we don’t care about the computational cost (as long as it is finite), then the “it is next token prediction” can’t preclude anything which “it produces a probability distribution over strings” doesn’t already preclude. As for the temperature, given any probability distribution over a discrete set, we can modify it by adding a temperature parameter. Just take the log of the probabilities according to the original probability distribution, scale them all by a factor (the inverse of the temperature, I think. Either that or the temperature, but I think it is the inverse of the temperature.), then exponentiate each of these, and then normalize to produce a probability distribution. So, the fact that they work by next token prediction, and have a temperature parameter, cannot imply any theoretical limitation that wouldn’t apply to any other way of expressing a probability distribution over strings, as far as discussing probability distributions in the abstract, over strings, rather than talking about computational processes that implement such probability distributions over strings. But also like, going between P(next token is x | initial string so far is y) and P(the string begins with z) , isn’t that computationally costly? Well, in one direction anyway. Because like, P(next token is x|string so far is y) = P(string begins with yx) / P(string begins with y) . Though, one might object to P(string starts with y) over P(string is y) ?
- abernard1 2y ago
  > Any probability distribution over strings can theoretically be factored into a product of such a “probability that next token is x given that the text so far is y”. And such a probability distribution would not generally understand concepts, efficient or otherwise. The P(next_token) is based upon the syntactical structure built via the model and some basic semantic distance that LLMs provide. They don't have enough conceptual power to reliably generate new facts and know that they are facts consistent with the model. That would be an internal representation system. The academic exercise here is similar to monads: "yes, any computed function f(x) can be expressed as a sufficiently pre-computed large lookup table." With LLMs we're dealing with approximate lookups due to lossy compression, but that's still what these prior probabilities are: lookup tables. Lookup tables are not smart, do not understand concepts, and they have little to no capacity to generate new results not sufficiently represented in the training set.
fshbbdssbbgdd 2y ago
> How could a system which produces a single next word based upon a likelihood and and a parameter called a "temperature" have a conceptual model underpinning it? Even theoretically? Could a creature that simply evolved to survive and reproduce possibly have a conceptual model underpinning it? Model training and evolution are very different processes, but they are both ways of optimizing a physical system. It may be the case that evolution can give rise to intelligence and model training can’t, but we need some argument to prove that.
- abernard1 2y ago
  > Could a creature that simply evolved to survive and reproduce possibly have a conceptual model underpinning it? Yes. Obviously. I can create plans and think on them. I can think without the need for an internal monologue or talking to myself. This has nothing to do with modalities either: I do not think through text. I use text when I do as a way of conveying the thoughts I already have. Anyone who claims we don't have the ability to form concepts in our head distinct from the medium in which they're transmitted is saying we're effectively ears, eyes, and skin. That the modality is what is important for intelligence. This is clearly false, and academic silliness aside, yes—emphatically—humans and intelligent agents have internal concepts and models of the world.
bubblyworld 2y ago
Transformer models have been shown to spontaneously form internal, predictive models of their input spaces. This is one of the most pervasive misunderstandings about LLMs (and other transformers) around. It is of course also true that the quality of these internal models depends a lot on the kind of task it is trained on. A GPT must be able to reproduce a huge swathe of human output, so the internal models it picks out would be those that are the most useful for that task, and might not include models of common mathematical tasks, for instance, unless they are common in the training set. Have a look at the OthelloGPT papers (can provide links if you're interested). This is one of the reasons people are so interested in them!
- brnt 2y ago
  > can provide links if you're interested Please do :)
  - persnickety 2y ago
    https://thegradient.pub/othello/ https://thegradient.pub/othello/
  - bubblyworld 2y ago
    Here's the paper on OthelloGPT's internal models I mentioned: https://arxiv.org/abs/2309.00941 https://arxiv.org/abs/2309.00941 The references in that paper are also good reading!
IanCal 2y ago
> How could a system which produces a single next word based upon a likelihood and and a parameter called a "temperature" have a conceptual model underpinning it? Even theoretically? You're limiting your view of their capabilities on the output format. > Not so with LLMs!! Generative LLMs do not have a prior concept available before they start emitting text. How do you establish that? What do you think of othellogpt? That seems to form an internal world model. > That the "temperature" can chaotically change the output as the tokens proceed Changing the temperature forcibly makes the model pick words it thinks fit worse. Of course it changes the output. It's like an improv game with someone shouting "CHANGE!". Let's make two tiny changes. One, let's tell a model to use the format <innerthought>askjdhas</innerthought> as the voice in their head, and <speak>blah</speak> for the output. Second, let's remove temperature and keep it at 0 so we're not playing a game where we force them to choose different words. Now what remains of the argument?
- abernard1 2y ago
  > You're limiting your view of their capabilities on the output format. The "generation" of strings is related to this output format. It's critical to how they work. Legerdemain has been performed to argue that that's irrelevant, and the real intelligence or concepts are sitting inside the network architecture of the trained model prior to generation. But if that were the case, generation could be done based upon the conceptual representation, and not a syntactical representation token by token. This is not currently the case with LLMs. I'd turn this question around: if my question is irrelevant, how would one go about building an effective real-world LLM that understand concepts and doesn't use likelihood lookups on a token-per-token basis, but instead generated directly from the conceptual basis? Such an argument, if it exists, would make me very happy. Please note, I understand that there are prior systems which do this. Generative zero-shot transformer models didn't adopt this approach because it's elegant, but because it is efficient to compute with large data sets and has useful efficacy in generating strings. Some are creative. Some are more "accurate." The temperature parameter can affect which of those cases it selects, if any. People have short memories, but the people who are both appreciative of LLMs as an engineering feat and critical of their claims of intelligence have been saying this for years. They've said that their token likelihood model is effective for seeing things well covered in the data set. They've been saying that due to the sparsity and structure of human language, large scale approximate compression ("curve fits") would be highly effective and efficient. They've been saying that due to the fact these are large scale fits of a data set, the models would eventually converge to something looking like the known knowledge they're trained on, and not exponentially accelerate in knowledge. All of these predictions have proved to be correct or looking highly likely at this point. Transformer-based LLMs are a neat algorithmic approach to curve fits. But they are curve fits. Things like cosine transforms in JPEGs, wavelet or Fourier reconstruction in CAT scans, audio signal reconstruction from basis functions are also approximate reconstruction models that function along these lines, albeit in a nice Euclidian space without the generative parts of a transformer. But it was precisely knowledge of how systems like that worked which allowed scientists to understand and predict the limitations of these systems a long time ago. Lots of money and fresh eyes have created a useful computation technique, but these insights have been forgotten. I hope—truly—that progress happens in this space. But the critiques stand and there would be lots to gain by a less-hyped acknowledgement of where we are with these models and the tradeoffs baked into them as a compromise for them to be useful.
l33tbro 2y ago
I've always loved your takes on AI. You should air them here a bit more.

GeneralMayhem 2y ago

The argument would be that that conceptual model is encoded in the intermediate-layer parameters of the model, in a different but analogous way to how it's encoded in the graph and chemical structure of your neurons.

abernard1 2y ago
I agree that's an argument. I would contend that argument is obviously false. If it were true, LLMs could multiply scalar numbers together trivially. It should be the easiest thing in the world for them. The network required to do that well is extremely small, the parameter sizes of these models are gigantic, and the textual expression is highly regular: multiplication is the simplest concept imaginable. That they cannot do that basic task implies to me that they have almost no conceptual understanding unless the fit is almost memorizable or the space is highly regular. That LLMs can't multiply numbers properly isn't surprising if they don't really understand concepts prior to emitting text. Where they do logical tasks, that can be done with minimal or no understanding, because syllogisms and logical formalisms are highly structured in text arguments.
- GaggiX 2y ago
  Multiplication requires O(n^2) complexity with the usual algorithm used by humans, LLMs have a constant amount of computation available and they are not really efficient machines for math evaluation. They can definitely evaluate unseen expressions and you train a neural network to learn how to do sums and multiplications, I have trained models on sums and they are able to do sums never seen during training, the model learns the algorithm just by giving it inputs and outputs.
- jdietrich 2y ago
  LLMs do contain conceptual representations and LLMs are capable of abstract reasoning. This is trivially provable by asking them to reason about something that is a) purely abstract and b) not in the training data, e.g. "All floots are gronks. Some gronks are klorps. Are any floots klorps?" Any of the leading LLMs will correctly answer questions of this type much more often than chance.
  - LetsGetTechnicl 2y ago
    That is not an example of a LLM being capable of abstract reasoning. Changing the question from "What is the capital of United States?" which is easily answerable to something completely abstract and "not in the training model" doesn't change that LLM's are just very advanced text prediction, and always will be. The nature of their design means they are incapable of AGI.
    - jdietrich 2y ago
      The question I gave is a literal textbook example of abstract reasoning. LLMs are just very advanced text prediction, but they are also provably capable of abstract reasoning. If you think that those statements are contradictory, I would encourage you to read up on the Bayesian hypotheses in cognitive science - it is highly plausible that our brains are also just very advanced prediction models.
      - nsagent 2y ago
nsagent 2y ago
Using Occam's razor, that is less probable than the model picking up on statistical regularities in human language, especially since that's what they are trained to do.
- mitthrowaway2 2y ago
  That's hard to conclude from Occam's razor here. Or, "statistical regularities" may have less explanatory power than you think, especially if the simplest statistical regularity is itself a fully predictive understanding of the concept of temperature.

Davidzheng 2y ago

It's only because you can essentially put the llms in a simulations that you can have this argument. We can imagine the human brain also in a simulation which we can replay over and over again and adjust various parameters of the physical brain to change the temperature. These sort of arguments can never distinguish between llm and humans.

gwervc 2y ago

> generative systems are basically always hallucinating: they do not have any concepts at all. That they are "right" as often as they are is a testament to the power of curve fitting and compression of basis functions in high dimensionality spaces It's refreshing to read someone who "got it". Sad that before my upvote the comment was grayed out. Any proponent of conceptual or other wishful/magical thinking shoud come with proofs, since it is the hypothesis that diverge from the definition of a LLM.

buu700 2y ago

On that point, I would dispute the premise that "it's impossible to have true language skills without implicitly having a representation of self and environment". I don't see any contradiction between the following two ideas: 1. LLMs inherently lack any form of consciousness, subjective experience, emotions, or will 2. A sufficiently advanced LLM with sufficient compute resources would perform on par with human intelligence at any given task, insofar as the task is applicable to LLMs

drdeca 2y ago

> I would argue it is (obviously) impossible the way the current implementation of models work. > How could a system which produces a single next word based upon a likelihood and and a parameter called a "temperature" have a conceptual model underpinning it? Even theoretically? Any probability distribution over strings can theoretically be factored into a product of such a “probability that next token is x given that the text so far is y”. Now, whether a probability distribution over strings can efficiently computed in this form, is another question. But, if we are being so theoretical that we don’t care about the computational cost (as long as it is finite), then the “it is next token prediction” can’t preclude anything which “it produces a probability distribution over strings” doesn’t already preclude. As for the temperature, given any probability distribution over a discrete set, we can modify it by adding a temperature parameter. Just take the log of the probabilities according to the original probability distribution, scale them all by a factor (the inverse of the temperature, I think. Either that or the temperature, but I think it is the inverse of the temperature.), then exponentiate each of these, and then normalize to produce a probability distribution. So, the fact that they work by next token prediction, and have a temperature parameter, cannot imply any theoretical limitation that wouldn’t apply to any other way of expressing a probability distribution over strings, as far as discussing probability distributions in the abstract, over strings, rather than talking about computational processes that implement such probability distributions over strings. But also like, going between P(next token is x | initial string so far is y) and P(the string begins with z) , isn’t that computationally costly? Well, in one direction anyway. Because like, P(next token is x|string so far is y) = P(string begins with yx) / P(string begins with y) . Though, one might object to P(string starts with y) over P(string is y) ?

abernard1 2y ago
> Any probability distribution over strings can theoretically be factored into a product of such a “probability that next token is x given that the text so far is y”. And such a probability distribution would not generally understand concepts, efficient or otherwise. The P(next_token) is based upon the syntactical structure built via the model and some basic semantic distance that LLMs provide. They don't have enough conceptual power to reliably generate new facts and know that they are facts consistent with the model. That would be an internal representation system. The academic exercise here is similar to monads: "yes, any computed function f(x) can be expressed as a sufficiently pre-computed large lookup table." With LLMs we're dealing with approximate lookups due to lossy compression, but that's still what these prior probabilities are: lookup tables. Lookup tables are not smart, do not understand concepts, and they have little to no capacity to generate new results not sufficiently represented in the training set.

fshbbdssbbgdd 2y ago

> How could a system which produces a single next word based upon a likelihood and and a parameter called a "temperature" have a conceptual model underpinning it? Even theoretically? Could a creature that simply evolved to survive and reproduce possibly have a conceptual model underpinning it? Model training and evolution are very different processes, but they are both ways of optimizing a physical system. It may be the case that evolution can give rise to intelligence and model training can’t, but we need some argument to prove that.

abernard1 2y ago
> Could a creature that simply evolved to survive and reproduce possibly have a conceptual model underpinning it? Yes. Obviously. I can create plans and think on them. I can think without the need for an internal monologue or talking to myself. This has nothing to do with modalities either: I do not think through text. I use text when I do as a way of conveying the thoughts I already have. Anyone who claims we don't have the ability to form concepts in our head distinct from the medium in which they're transmitted is saying we're effectively ears, eyes, and skin. That the modality is what is important for intelligence. This is clearly false, and academic silliness aside, yes—emphatically—humans and intelligent agents have internal concepts and models of the world.

bubblyworld 2y ago

Transformer models have been shown to spontaneously form internal, predictive models of their input spaces. This is one of the most pervasive misunderstandings about LLMs (and other transformers) around. It is of course also true that the quality of these internal models depends a lot on the kind of task it is trained on. A GPT must be able to reproduce a huge swathe of human output, so the internal models it picks out would be those that are the most useful for that task, and might not include models of common mathematical tasks, for instance, unless they are common in the training set. Have a look at the OthelloGPT papers (can provide links if you're interested). This is one of the reasons people are so interested in them!

brnt 2y ago
> can provide links if you're interested Please do :)
- persnickety 2y ago
  https://thegradient.pub/othello/ https://thegradient.pub/othello/
- bubblyworld 2y ago
  Here's the paper on OthelloGPT's internal models I mentioned: https://arxiv.org/abs/2309.00941 https://arxiv.org/abs/2309.00941 The references in that paper are also good reading!

IanCal 2y ago

> How could a system which produces a single next word based upon a likelihood and and a parameter called a "temperature" have a conceptual model underpinning it? Even theoretically? You're limiting your view of their capabilities on the output format. > Not so with LLMs!! Generative LLMs do not have a prior concept available before they start emitting text. How do you establish that? What do you think of othellogpt? That seems to form an internal world model. > That the "temperature" can chaotically change the output as the tokens proceed Changing the temperature forcibly makes the model pick words it thinks fit worse. Of course it changes the output. It's like an improv game with someone shouting "CHANGE!". Let's make two tiny changes. One, let's tell a model to use the format <innerthought>askjdhas</innerthought> as the voice in their head, and <speak>blah</speak> for the output. Second, let's remove temperature and keep it at 0 so we're not playing a game where we force them to choose different words. Now what remains of the argument?

abernard1 2y ago
> You're limiting your view of their capabilities on the output format. The "generation" of strings is related to this output format. It's critical to how they work. Legerdemain has been performed to argue that that's irrelevant, and the real intelligence or concepts are sitting inside the network architecture of the trained model prior to generation. But if that were the case, generation could be done based upon the conceptual representation, and not a syntactical representation token by token. This is not currently the case with LLMs. I'd turn this question around: if my question is irrelevant, how would one go about building an effective real-world LLM that understand concepts and doesn't use likelihood lookups on a token-per-token basis, but instead generated directly from the conceptual basis? Such an argument, if it exists, would make me very happy. Please note, I understand that there are prior systems which do this. Generative zero-shot transformer models didn't adopt this approach because it's elegant, but because it is efficient to compute with large data sets and has useful efficacy in generating strings. Some are creative. Some are more "accurate." The temperature parameter can affect which of those cases it selects, if any. People have short memories, but the people who are both appreciative of LLMs as an engineering feat and critical of their claims of intelligence have been saying this for years. They've said that their token likelihood model is effective for seeing things well covered in the data set. They've been saying that due to the sparsity and structure of human language, large scale approximate compression ("curve fits") would be highly effective and efficient. They've been saying that due to the fact these are large scale fits of a data set, the models would eventually converge to something looking like the known knowledge they're trained on, and not exponentially accelerate in knowledge. All of these predictions have proved to be correct or looking highly likely at this point. Transformer-based LLMs are a neat algorithmic approach to curve fits. But they are curve fits. Things like cosine transforms in JPEGs, wavelet or Fourier reconstruction in CAT scans, audio signal reconstruction from basis functions are also approximate reconstruction models that function along these lines, albeit in a nice Euclidian space without the generative parts of a transformer. But it was precisely knowledge of how systems like that worked which allowed scientists to understand and predict the limitations of these systems a long time ago. Lots of money and fresh eyes have created a useful computation technique, but these insights have been forgotten. I hope—truly—that progress happens in this space. But the critiques stand and there would be lots to gain by a less-hyped acknowledgement of where we are with these models and the tradeoffs baked into them as a compromise for them to be useful.

l33tbro 2y ago

I've always loved your takes on AI. You should air them here a bit more.