7 ms·

It's a surprise to see a paper actually try to solve the problem of modelling thought via language. Nevertheless, it begins with far too many hedges: > By sca

It's a surprise to see a paper actually try to solve the problem of modelling thought via language. Nevertheless, it begins with far too many hedges: > By scaling to even larger datasets and neural networks, LLMs appeared to learn not only the structure of language, but capacities for some kinds of thinking There's two hypotheses for how LLMs generate apparently "thought-expressing" outputs: Hyp1 -- it's sampling from similar text which is distributed so-as-to-express a thought by some agent; Hyp2 -- it has the capacity to form that thought. It is absolutely trivial to show Hyp2 is false: > Current LLMs can produce impressive results on a set of linguistic inputs and then fail completely on others that make trivial alterations to the same underlying domain. Indeed: because there're no relevant prior cases to sample from in that case. > These issues make it difficult to evaluate whether LLMs have acquired cognitive capacities such as social reasoning and theory of mind It doesnt. It's trivial: the disproof lies one sentence above. Its just that many don't like the answer. Such capacities survive trivial permutations -- LLMs do not. So Hypothesis-2 is clearly false.

rytill 3y ago
I don't think you really disproved anything. You're just saying another hypothesis. Often, LLMs produce impressive results on domains that aren't in the training set.
- sgt101 3y ago
  >LLMs produce impressive results on domains that aren't in the training set. How do we know? Who knows what they're trained on?
famouswaffles 3y ago
>It is absolutely trivial to show Hyp2 is false No it's not > Current LLMs can produce impressive results on a set of linguistic inputs and then fail completely on others that make trivial alterations to the same underlying domain. >Indeed: because there're no relevant prior cases to sample from in that case. That's not what that tells us. Humans have weird failure modes that look absurd outside the context of evolutionary biology (some still look absurd) and that don't speak to any lack or presence of intelligence or complex thought. Not sure why it's so hard to grasp that LLMs are bound to have odd failure modes regardless of the above. and trivial here is relative. In my experience, "trivial" often turns out to be trivial in the way a person may not pay close attention to and be similarly tricked. For instance, GPT-4 might solve a classic puzzle correctly then fail the same puzzle subtlety changed. I've found more often than not, simply changing names of variables in the puzzle to something completely different can get it to solve the changed puzzle. It takes memory shortcuts but can be pulled out of that. LLMs have failure modes that look like human failure modes too.
- mjburgess 3y ago
  The "failure modes" in humans do not show we lack the capacity. Eg., do you have capacity to reason about physics? Well if you're extremely drunk, less so. But not if I permute the name of the object. > I've found more often than not, simply changing names of variables Yes, lol --- why do you think that is? Because in the digitised dataset of "everything ever written" those names correspond to places in that dataset that can be sampled from by the LLM. Showing Hyp1 to be the case. P(Hyp1| ChangeNameMakesDifference) >>>>>> P(Hyp2|ChangeNameMakesDifference) To such a degree that the latter is vanishingly close to zero.
  - famouswaffles 3y ago
    >The "failure modes" in humans do not show we lack the capacity. Then they don't in LLMs too >Yes, lol --- why do you think that is? Being able to solve a changed common puzzle but also with different names than it would ever see in training is not an indication of a lack of ability lol. and changing names isn't the only way to get it out of memory, just the easiest/most straightforward. You can converse it out of there too but that doesn't work as often.
    - mjburgess 3y ago
      > Then they don't in LLMs too LLMs don't get drunk . If a child answers questions from a book of answers then they'll appear to understand the domain insofar as those questions appear. They do not. They will fail to answer questions under, eg., permutations of words (say, a question asks about "norepinephrine" but the book only contains "noradrenaline" etc.). Insofar as a human cannot answer questions under trivial linguistic permutations then they too do not understand the domain. But these are not the kinds of failures experienced with those who have some capacity, eg., for counter-factual reasoning about their environment's physics. In those people it is environmental illusion and cognitive impairment -- not trivial permutations of phrasing which lead to catastrophic loss of apparent understanding. Cognitive impairment = reasoning machine is broken Environmental illusion = data is ambigious and actions cannto resolve it These "failure modes" are expected if you actually have the relevant capacity.
redox99 3y ago
If it's "absolutely trivial" to show that LLMs don't have the capacity to form thought, then please publish a paper proving that. So all the "stupid" people studying LLMs that can't come up with such trivial proofs can move on to other stuff.
- parpfish 3y ago
  "I have a truly marvelous demonstration that LLMs don't have the capacity to form thought which this margin is too narrow to contain."
- mjburgess 3y ago
  You may wish to read the paper above. But if you want a quick proof: 1. A thought is a representation of a situation 2. A representation generates entailments of that situation 3. Language is many-to-one translation from these representations to symbols 4. Understanding language is reversing these symbols into thoughts (ie., reprs) So, 5. If agent A understands sentence X then A forms the relevant representation of X. 6. If agent has a representation it can state entailments of S (eg., counter-facutals). Now, split X into Xc = "canonical descriptions of S" and trivial permutations Xp. (st. distribution of Xc,Xp is low, but the tokens of Xp are common) Form entailments of X, say Y -- sentences that are cannonically implied by the truth of X. 7. If the LLM understood that X entails Y, it would be via constructing the repr S -- which entails S regardless of which sentence in X was used. 8. Train an LLM on Xc and it's accuracy on judging Y entailed by Xp is random. 9. Since using Xp sentences cause it to fail, it does not predict Y via S. QED. And we can say, 1. Appearing to judge Y entailed-by X is possible via simple sampling of (X, Y) in historical cases. 2. LLMs are just such a sampling. so, 3. +Inference to the best explanation: 4. LLMs sample historical cases rather than form representations. Incidentally, "sampling of historical cases" is already something we knew -- so this entire argument is basically unnecessary. And only necessary because PhDs have been turned into start-up hype men.
  - lostmsu 3y ago
    > Train an LLM on Xc and it's accuracy on judging Y entailed by Xp is random. Why? This is obviously wrong in general case. For that to be true Xp and Xc has to have no statistical relationship whatsoever, which statistically is virtually impossible.
fiso64 3y ago
Don't try to ham-fist scientific sounding wording into your (very unscientific) argument. This is not a disproof of anything because you failed to define what it means to have the ability to form rational thoughts. With a definition, you would then wanna prove this for humans as a sanity check: Do we never make stupid mistakes? Ok, we make fewer of those than LLMs. Then what is the threshold for accuracy after which you consider a system to be intelligent? Do all humans pass that threshold, or do kids or people with a lower than average IQ fail?
- mdp2021 3y ago
  > humans There is intelligent thought and action, and there is unintelligent thought and action. Intelligent is that "which checked" (intus-legere); the other, the """impulsive""", is not.
- mjburgess 3y ago
  This entire paper is written as a disproof of the distributional hypothesis. If you want to understand why it's a profoundly unhelpful pseudoscientific idea, this paper is a good start. The test for a capacity C in a system1 has nothing to do with proxy measures of that capacity in system2. The capacity for an oven to cook food may be measured by how much smoke it lets of when burning -- but no amount of "smoke" establishes that a dry ice machine can cook. This type of "engineering thinking" is pseudoscience.
  - fiso64 3y ago
    >The capacity for an oven to cook food may be measured by how much smoke it lets of when burning -- but no amount of "smoke" establishes that a dry ice machine can cook. You seem to be talking past me, as nowhere did I claim that LLMs are intelligent. That's the point – Unlike you I do not claim to be able to prove or disprove this. I argue that your comment is the one that is pseudoscientific because you didn't provide (even a semblance of) a rigorous definition of intelligence.
jbay808 3y ago
> It is absolutely trivial to show Hyp2 is false To investigate precisely this question in a clear and unambiguous way, I trained an LLM from scratch to sort lists of numbers. It learned to sort them correctly, and the entropy is such that it's absolutely impossible that it could have done this by Hyp1 (sampling from similar text in the training set). https://jbconsulting.substack.com/p/its-not-just-statistics-gpt-4-does https://jbconsulting.substack.com/p/its-not-just-statistics-... Now, there is room to argue that it applies a world-model when given lists of numbers with a hidden logical structure, but not when given lists of words with a hidden logical structure, but I think the ball is in your court to make that argument. (And to a transformer, it only ever sees lists of numbers anyway).
- mjburgess 3y ago
  So this is a really good starting point -- but you havent formulated any hypotheses that can be tested. You've just looked at the graph and "reckoned something". Formally, what hypotheses are you comparing? What do you think the specific hypothesis of the "AI = stats" person is? It isnt that the NN literally remembers data tokens, right? In any case: The issue with forcing NNs to model mathematical features is that the structure of the data itself has those properties. So the distributional hypothesis is true for sorting ordinals. But it's really obviously false for natural language. The properties of the world are not the properties of word order... being red isnt "red follows words like...".
  - jbay808 3y ago
    > you havent formulated any hypotheses that can be tested. You've just looked at the graph and "reckoned something" Let's not be so hasty. I think I do put it as clearly as possible. I'm comparing essentially your Hyp1 and Hyp2, where Hyp1 (aka the stochastic parrot) is expressed a little bit more clearly as the LLM is learning an n-gram that produces correct sorts through rote memorization of statistical correlations in the training data, like that sorted lists tend to start with '0', end with '99', and increase monotonically; and Hyp2 is that the LLM's training molds it into representing an actual sorting algorithm that would correctly generalize to any input list. > But it's really obviously false for natural language. The properties of the world are not the properties of word order... being red isnt "red follows words like..." This is not really obviously false. Yes, being red isn't "red follows words like...". But a word order should still map to properties of the world, especially if those words are to be meaningful to a listener. Being red is "a surface reflects or transmits most of the light in the 600-800 nm spectrum and absorbs most of the rest". Of course, it won't do to just echo those tokens; once you've nailed down the concept of "red", you need to make sure that concepts like "reflects", "light", and "spectrum" are represented as well. It's an open question as to whether this sort of knowledge graph can be properly bootstrapped from a large volume of text descriptions, but I am strongly inclined to believe it can. If you dismiss it outright you're just begging the question.
TeMPOraL 3y ago
> There's two hypotheses for how LLMs generate apparently "thought-expressing" outputs: Hyp1 -- it's sampling from similar text which is distributed so-as-to-express a thought by some agent; Hyp2 -- it has the capacity to form that thought. There's also another hypothesis: Hyp3 -- that Hyp1 and Hyp2 converge as the LLM is scaled up (more training data, more dimensions in the latent space), and in the limit become equivalent.
- mjburgess 3y ago
  They're indistinguishable via naive measurement (prompting) if the LLM can sample from all possible data: there's a very large infinity of (Q, A, time) triples (ie., it's real-valued). But it cannot, since most of those are in the future.
bryan0 3y ago
Failing on "trivial alterations to the same underlying domain" is a not a disproof of thought. Your argument also implies hyp1 and 2 are exclusive, clearly both can be true, and in fact must be true, unless you are claiming that you do not "sample" from similar language to express your own thoughts? Where does your language come from then, if not learning from previous experience?
- anankaie 3y ago
  While I agree with you on the relation of GP's Hyp1 and Hyp2, you are making an unfounded assumption of a sampling process being necessary to perform human speech. I do not believe we have the understanding of how thought is represented in the human brain to make that judgement. In other words, just because sampling from a distribution can produce human-like text does not mean that it is the only way to do that, and thus that it must be the way that humans produce text, spoken or written.
  - bryan0 3y ago
    We might be talking about 2 different things. I was referring to the backwards learning pass and you seem to be referring to the forward inference pass, but what is an alternative to learning (or producing) text which does not involve sampling from some larger space? (Also I’m not a statistician so I’m not sure if these are technically “distributions”)
mirekrusin 3y ago
"Trivial to show" as in it's trivial to show that addition on uint8 doesn't work ie. 250+250?
canjobear 3y ago
> it's sampling from similar text which is distributed so-as-to-express a thought by some agent; Your hypotheses 1 and 2 are not so different when you consider that the similarity function used to match text in the training data must be highly nontrivial. If it were not, then things like GPT-3 would have been possible a long time ago. As a concrete example, LLMs can do decent reasoning entirely in rot13; the relevant rot13'ed text is likely very rare in their training data. The fact that the similarity function can "see through" rot13 means that it can in principle include nontrivial computations.

rytill 3y ago

I don't think you really disproved anything. You're just saying another hypothesis. Often, LLMs produce impressive results on domains that aren't in the training set.

sgt101 3y ago
>LLMs produce impressive results on domains that aren't in the training set. How do we know? Who knows what they're trained on?

famouswaffles 3y ago

>It is absolutely trivial to show Hyp2 is false No it's not > Current LLMs can produce impressive results on a set of linguistic inputs and then fail completely on others that make trivial alterations to the same underlying domain. >Indeed: because there're no relevant prior cases to sample from in that case. That's not what that tells us. Humans have weird failure modes that look absurd outside the context of evolutionary biology (some still look absurd) and that don't speak to any lack or presence of intelligence or complex thought. Not sure why it's so hard to grasp that LLMs are bound to have odd failure modes regardless of the above. and trivial here is relative. In my experience, "trivial" often turns out to be trivial in the way a person may not pay close attention to and be similarly tricked. For instance, GPT-4 might solve a classic puzzle correctly then fail the same puzzle subtlety changed. I've found more often than not, simply changing names of variables in the puzzle to something completely different can get it to solve the changed puzzle. It takes memory shortcuts but can be pulled out of that. LLMs have failure modes that look like human failure modes too.

mjburgess 3y ago
The "failure modes" in humans do not show we lack the capacity. Eg., do you have capacity to reason about physics? Well if you're extremely drunk, less so. But not if I permute the name of the object. > I've found more often than not, simply changing names of variables Yes, lol --- why do you think that is? Because in the digitised dataset of "everything ever written" those names correspond to places in that dataset that can be sampled from by the LLM. Showing Hyp1 to be the case. P(Hyp1| ChangeNameMakesDifference) >>>>>> P(Hyp2|ChangeNameMakesDifference) To such a degree that the latter is vanishingly close to zero.
- famouswaffles 3y ago
  >The "failure modes" in humans do not show we lack the capacity. Then they don't in LLMs too >Yes, lol --- why do you think that is? Being able to solve a changed common puzzle but also with different names than it would ever see in training is not an indication of a lack of ability lol. and changing names isn't the only way to get it out of memory, just the easiest/most straightforward. You can converse it out of there too but that doesn't work as often.
  - mjburgess 3y ago
    > Then they don't in LLMs too LLMs don't get drunk . If a child answers questions from a book of answers then they'll appear to understand the domain insofar as those questions appear. They do not. They will fail to answer questions under, eg., permutations of words (say, a question asks about "norepinephrine" but the book only contains "noradrenaline" etc.). Insofar as a human cannot answer questions under trivial linguistic permutations then they too do not understand the domain. But these are not the kinds of failures experienced with those who have some capacity, eg., for counter-factual reasoning about their environment's physics. In those people it is environmental illusion and cognitive impairment -- not trivial permutations of phrasing which lead to catastrophic loss of apparent understanding. Cognitive impairment = reasoning machine is broken Environmental illusion = data is ambigious and actions cannto resolve it These "failure modes" are expected if you actually have the relevant capacity.

redox99 3y ago

If it's "absolutely trivial" to show that LLMs don't have the capacity to form thought, then please publish a paper proving that. So all the "stupid" people studying LLMs that can't come up with such trivial proofs can move on to other stuff.

parpfish 3y ago
"I have a truly marvelous demonstration that LLMs don't have the capacity to form thought which this margin is too narrow to contain."
mjburgess 3y ago
You may wish to read the paper above. But if you want a quick proof: 1. A thought is a representation of a situation 2. A representation generates entailments of that situation 3. Language is many-to-one translation from these representations to symbols 4. Understanding language is reversing these symbols into thoughts (ie., reprs) So, 5. If agent A understands sentence X then A forms the relevant representation of X. 6. If agent has a representation it can state entailments of S (eg., counter-facutals). Now, split X into Xc = "canonical descriptions of S" and trivial permutations Xp. (st. distribution of Xc,Xp is low, but the tokens of Xp are common) Form entailments of X, say Y -- sentences that are cannonically implied by the truth of X. 7. If the LLM understood that X entails Y, it would be via constructing the repr S -- which entails S regardless of which sentence in X was used. 8. Train an LLM on Xc and it's accuracy on judging Y entailed by Xp is random. 9. Since using Xp sentences cause it to fail, it does not predict Y via S. QED. And we can say, 1. Appearing to judge Y entailed-by X is possible via simple sampling of (X, Y) in historical cases. 2. LLMs are just such a sampling. so, 3. +Inference to the best explanation: 4. LLMs sample historical cases rather than form representations. Incidentally, "sampling of historical cases" is already something we knew -- so this entire argument is basically unnecessary. And only necessary because PhDs have been turned into start-up hype men.
- lostmsu 3y ago
  > Train an LLM on Xc and it's accuracy on judging Y entailed by Xp is random. Why? This is obviously wrong in general case. For that to be true Xp and Xc has to have no statistical relationship whatsoever, which statistically is virtually impossible.

fiso64 3y ago

Don't try to ham-fist scientific sounding wording into your (very unscientific) argument. This is not a disproof of anything because you failed to define what it means to have the ability to form rational thoughts. With a definition, you would then wanna prove this for humans as a sanity check: Do we never make stupid mistakes? Ok, we make fewer of those than LLMs. Then what is the threshold for accuracy after which you consider a system to be intelligent? Do all humans pass that threshold, or do kids or people with a lower than average IQ fail?

mdp2021 3y ago
> humans There is intelligent thought and action, and there is unintelligent thought and action. Intelligent is that "which checked" (intus-legere); the other, the """impulsive""", is not.
mjburgess 3y ago
This entire paper is written as a disproof of the distributional hypothesis. If you want to understand why it's a profoundly unhelpful pseudoscientific idea, this paper is a good start. The test for a capacity C in a system1 has nothing to do with proxy measures of that capacity in system2. The capacity for an oven to cook food may be measured by how much smoke it lets of when burning -- but no amount of "smoke" establishes that a dry ice machine can cook. This type of "engineering thinking" is pseudoscience.
- fiso64 3y ago
  >The capacity for an oven to cook food may be measured by how much smoke it lets of when burning -- but no amount of "smoke" establishes that a dry ice machine can cook. You seem to be talking past me, as nowhere did I claim that LLMs are intelligent. That's the point – Unlike you I do not claim to be able to prove or disprove this. I argue that your comment is the one that is pseudoscientific because you didn't provide (even a semblance of) a rigorous definition of intelligence.

jbay808 3y ago

> It is absolutely trivial to show Hyp2 is false To investigate precisely this question in a clear and unambiguous way, I trained an LLM from scratch to sort lists of numbers. It learned to sort them correctly, and the entropy is such that it's absolutely impossible that it could have done this by Hyp1 (sampling from similar text in the training set). https://jbconsulting.substack.com/p/its-not-just-statistics-gpt-4-does https://jbconsulting.substack.com/p/its-not-just-statistics-... Now, there is room to argue that it applies a world-model when given lists of numbers with a hidden logical structure, but not when given lists of words with a hidden logical structure, but I think the ball is in your court to make that argument. (And to a transformer, it only ever sees lists of numbers anyway).

mjburgess 3y ago
So this is a really good starting point -- but you havent formulated any hypotheses that can be tested. You've just looked at the graph and "reckoned something". Formally, what hypotheses are you comparing? What do you think the specific hypothesis of the "AI = stats" person is? It isnt that the NN literally remembers data tokens, right? In any case: The issue with forcing NNs to model mathematical features is that the structure of the data itself has those properties. So the distributional hypothesis is true for sorting ordinals. But it's really obviously false for natural language. The properties of the world are not the properties of word order... being red isnt "red follows words like...".
- jbay808 3y ago
  > you havent formulated any hypotheses that can be tested. You've just looked at the graph and "reckoned something" Let's not be so hasty. I think I do put it as clearly as possible. I'm comparing essentially your Hyp1 and Hyp2, where Hyp1 (aka the stochastic parrot) is expressed a little bit more clearly as the LLM is learning an n-gram that produces correct sorts through rote memorization of statistical correlations in the training data, like that sorted lists tend to start with '0', end with '99', and increase monotonically; and Hyp2 is that the LLM's training molds it into representing an actual sorting algorithm that would correctly generalize to any input list. > But it's really obviously false for natural language. The properties of the world are not the properties of word order... being red isnt "red follows words like..." This is not really obviously false. Yes, being red isn't "red follows words like...". But a word order should still map to properties of the world, especially if those words are to be meaningful to a listener. Being red is "a surface reflects or transmits most of the light in the 600-800 nm spectrum and absorbs most of the rest". Of course, it won't do to just echo those tokens; once you've nailed down the concept of "red", you need to make sure that concepts like "reflects", "light", and "spectrum" are represented as well. It's an open question as to whether this sort of knowledge graph can be properly bootstrapped from a large volume of text descriptions, but I am strongly inclined to believe it can. If you dismiss it outright you're just begging the question.

TeMPOraL 3y ago

> There's two hypotheses for how LLMs generate apparently "thought-expressing" outputs: Hyp1 -- it's sampling from similar text which is distributed so-as-to-express a thought by some agent; Hyp2 -- it has the capacity to form that thought. There's also another hypothesis: Hyp3 -- that Hyp1 and Hyp2 converge as the LLM is scaled up (more training data, more dimensions in the latent space), and in the limit become equivalent.

mjburgess 3y ago
They're indistinguishable via naive measurement (prompting) if the LLM can sample from all possible data: there's a very large infinity of (Q, A, time) triples (ie., it's real-valued). But it cannot, since most of those are in the future.

bryan0 3y ago

Failing on "trivial alterations to the same underlying domain" is a not a disproof of thought. Your argument also implies hyp1 and 2 are exclusive, clearly both can be true, and in fact must be true, unless you are claiming that you do not "sample" from similar language to express your own thoughts? Where does your language come from then, if not learning from previous experience?

anankaie 3y ago
While I agree with you on the relation of GP's Hyp1 and Hyp2, you are making an unfounded assumption of a sampling process being necessary to perform human speech. I do not believe we have the understanding of how thought is represented in the human brain to make that judgement. In other words, just because sampling from a distribution can produce human-like text does not mean that it is the only way to do that, and thus that it must be the way that humans produce text, spoken or written.
- bryan0 3y ago
  We might be talking about 2 different things. I was referring to the backwards learning pass and you seem to be referring to the forward inference pass, but what is an alternative to learning (or producing) text which does not involve sampling from some larger space? (Also I’m not a statistician so I’m not sure if these are technically “distributions”)

mirekrusin 3y ago

"Trivial to show" as in it's trivial to show that addition on uint8 doesn't work ie. 250+250?

canjobear 3y ago

> it's sampling from similar text which is distributed so-as-to-express a thought by some agent; Your hypotheses 1 and 2 are not so different when you consider that the similarity function used to match text in the training data must be highly nontrivial. If it were not, then things like GPT-3 would have been possible a long time ago. As a concrete example, LLMs can do decent reasoning entirely in rot13; the relevant rot13'ed text is likely very rare in their training data. The fact that the similarity function can "see through" rot13 means that it can in principle include nontrivial computations.