6 ms·

Think of it this way: They create plausible patterns of language that seem (to humans) similar to human-generated language. That is what they are "good at". Im

Think of it this way: They create plausible patterns of language that seem (to humans) similar to human-generated language. That is what they are "good at". Imagine an "AI" that "knows" about rhyme and meter and can generate a seemingly infinite collection of catchy songs that follow the II V I chord progression, only the sequence of words doesn't necessarily make any sense. LLMs are a more powerful version of that kind of thing that does put the words in a sequence that feels like natural human language. Since they are trained on information, the sequences chosen often seem informational, even though they are just patterns. It's really a lot like autocomplete. In other words, a lot of useful information may accidentally be encoded into the model, but so is a lot of stuff that looks/feels similar but is nonsense.

erwald 4y ago
I think I mostly agree with you, but I think this framing is a bit misleading. On autocomplete, I'll just lazily quote the relevant part from my post: It’s completely true that LLMs are trained on next-token prediction (although some, like ChatGPT, are then additionally trained using reinforcement learning with human feedback). It’s also completely true that this fact profoundly influences the texts they generate. So I don’t think it’s unreasonable to call LLMs autocomplete engines or to emphasise next-token prediction. But I think it’s subtly misleading: - Though LLMs were trained to optimise success on next-token prediction, that is not necessarily what they do. We don’t know what it is they do. The training process reinforces behaviours/heuristics in the model that tend to cause it to make better next-token predictions on in-distribution data. This does not mean that those behaviours/heuristics are fundamentally “about” optimising next-token prediction, especially when the model encounters out-of-distribution data. - The usual example here is human evolution. Humans were shaped by a process that optimised for reproductive fitness. This gave us a bundle of drives such as family kinship, prestige and sexual pleasure – drives that aren’t fundamentally about optimising for reproduction, which becomes evident as we enter a new environment – one with contraceptives, say. - Optimising for a task for which intelligence is useful encourages the optimised thing to become more intelligent. Sam Altman gave expression to this last week when he wrote, “Language models just being programmed to try to predict the next word is true, but it’s not the dunk some people think it is. Animals, including us, are just programmed to try to survive and reproduce, and yet amazingly complex and beautiful stuff comes from it.” - The forms of intelligence that are useful in doing next-token prediction are different from those that are useful in human reproduction, but I think there’s a considerable overlap, as (1) some fundamental abilities, for example using and applying concepts, just seem very broadly useful and (2) the data LLMs are trained on are written by humans, for humans and often about humans and things that matter to us. I think the "they're just autocomplete" take also hides other properties of LLMs, like them seeming to (as mentioned in another comment, and in the post) contain and use world models, and being able to learn general algorithms.
naasking 4y ago
> Since they are trained on information, the sequences chosen often seem informational, even though they are just patterns. The point is is that dismissing it because it's "just patterns" ignores the fact that nobody has proven that human cognition isn't also "just patterns". If it is, then that undermines the whole justification for the dismissal. So the dismissal assumes the conclusion until that evidence is presented.
- slowmovintarget 4y ago
  All the arguments for LLMs actually having "understanding" are of the form "I want to believe" or "you can't prove they're not." Chomsky's point that we have no basis for judging LLMs as cognition machines because we don't know what cognition is, is correct. Building these machines may produce useful tools, but it also doesn't advance studies of cognition, either. We do know one characteristic of cognition as it occurs in nature. There seem to be natural affinities to distinguish the real and true from the hallucination. Perhaps that comes from persistence and physicality. How do we give that to ChatGPT?
  - naasking 4y ago
    > Chomsky's point that we have no basis for judging LLMs as cognition machines because we don't know what cognition is, is correct. That's not Chomsky's point at all. He's saying they are emphatically not understanding or intelligent in any way, and that's not justified. > Building these machines may produce useful tools, but it also doesn't advance studies of cognition, either. I couldn't disagree more. We have here apparently intelligent systems completely unlike naturally occurring ones. IMO, you'd have to be crazy to think that comparing and contrasting the properties of these systems won't provide any valuable data about the nature of intelligence, perception and cognition as a whole. In fact, we're already seeing papers discussing how LLMs basically refute Chomsky's whole model of language [1]. [1] https://lingbuzz.net/lingbuzz/007180 https://lingbuzz.net/lingbuzz/007180
    - naasking 4y ago
      To clarify, by "completely unlike naturally occurring ones", I mean their origin and training set is completely unlike them, not that their operation is mechanistically different. We don't really know that yet.

erwald 4y ago

I think I mostly agree with you, but I think this framing is a bit misleading. On autocomplete, I'll just lazily quote the relevant part from my post: It’s completely true that LLMs are trained on next-token prediction (although some, like ChatGPT, are then additionally trained using reinforcement learning with human feedback). It’s also completely true that this fact profoundly influences the texts they generate. So I don’t think it’s unreasonable to call LLMs autocomplete engines or to emphasise next-token prediction. But I think it’s subtly misleading: - Though LLMs were trained to optimise success on next-token prediction, that is not necessarily what they do. We don’t know what it is they do. The training process reinforces behaviours/heuristics in the model that tend to cause it to make better next-token predictions on in-distribution data. This does not mean that those behaviours/heuristics are fundamentally “about” optimising next-token prediction, especially when the model encounters out-of-distribution data. - The usual example here is human evolution. Humans were shaped by a process that optimised for reproductive fitness. This gave us a bundle of drives such as family kinship, prestige and sexual pleasure – drives that aren’t fundamentally about optimising for reproduction, which becomes evident as we enter a new environment – one with contraceptives, say. - Optimising for a task for which intelligence is useful encourages the optimised thing to become more intelligent. Sam Altman gave expression to this last week when he wrote, “Language models just being programmed to try to predict the next word is true, but it’s not the dunk some people think it is. Animals, including us, are just programmed to try to survive and reproduce, and yet amazingly complex and beautiful stuff comes from it.” - The forms of intelligence that are useful in doing next-token prediction are different from those that are useful in human reproduction, but I think there’s a considerable overlap, as (1) some fundamental abilities, for example using and applying concepts, just seem very broadly useful and (2) the data LLMs are trained on are written by humans, for humans and often about humans and things that matter to us. I think the "they're just autocomplete" take also hides other properties of LLMs, like them seeming to (as mentioned in another comment, and in the post) contain and use world models, and being able to learn general algorithms.

naasking 4y ago

> Since they are trained on information, the sequences chosen often seem informational, even though they are just patterns. The point is is that dismissing it because it's "just patterns" ignores the fact that nobody has proven that human cognition isn't also "just patterns". If it is, then that undermines the whole justification for the dismissal. So the dismissal assumes the conclusion until that evidence is presented.

slowmovintarget 4y ago
All the arguments for LLMs actually having "understanding" are of the form "I want to believe" or "you can't prove they're not." Chomsky's point that we have no basis for judging LLMs as cognition machines because we don't know what cognition is, is correct. Building these machines may produce useful tools, but it also doesn't advance studies of cognition, either. We do know one characteristic of cognition as it occurs in nature. There seem to be natural affinities to distinguish the real and true from the hallucination. Perhaps that comes from persistence and physicality. How do we give that to ChatGPT?
- naasking 4y ago
  > Chomsky's point that we have no basis for judging LLMs as cognition machines because we don't know what cognition is, is correct. That's not Chomsky's point at all. He's saying they are emphatically not understanding or intelligent in any way, and that's not justified. > Building these machines may produce useful tools, but it also doesn't advance studies of cognition, either. I couldn't disagree more. We have here apparently intelligent systems completely unlike naturally occurring ones. IMO, you'd have to be crazy to think that comparing and contrasting the properties of these systems won't provide any valuable data about the nature of intelligence, perception and cognition as a whole. In fact, we're already seeing papers discussing how LLMs basically refute Chomsky's whole model of language [1]. [1] https://lingbuzz.net/lingbuzz/007180 https://lingbuzz.net/lingbuzz/007180
  - naasking 4y ago
    To clarify, by "completely unlike naturally occurring ones", I mean their origin and training set is completely unlike them, not that their operation is mechanistically different. We don't really know that yet.