7 ms·

> If you want to be really good, you have to understand what’s being said. That’s the only way. This is simply not true. Predicting the next letter or word, or

> If you want to be really good, you have to understand what’s being said. That’s the only way. This is simply not true. Predicting the next letter or word, or id you abstract it away from things that mean something to you, like the next color of a block in a long chain of colored blocks. You would realize that all we are doing is using statistics to predict what the next item might be. There simply is no need or requirement for the system doing the prediction to understand.

continuational 3y ago
I think the most useful definition of understanding something is that you can explain it and use it in context. ChatGPT routinely does both.
- jstummbillig 3y ago
  And while AI gets better and better and we will remain as touchy as ever about abstract concepts that make us oh so human, how about we say it just can't be understanding, unless a human does, eh, it.
- BobaFloutist 3y ago
  Someone sufficiently fast and skilled at googling can explain and use in context a lot of things that they don't really properly understand. So unless you're saying that the composite of the googler and of google understand something that neither does individually, your definition has some holes.
  - continuational 3y ago
    This is a variation of the Chinese room argument. If you consider understanding an observable property, then the Chinese room in aggregate displays understanding of Chinese. Would you say that humans understand nothing, because atoms don't understand anything, and we're made up of atoms?
    - BobaFloutist 3y ago
      I would say that there is a stronger consensus that a human being can be reasonably described as a single entity than a human being using a reference resource. A more apt comparison to my mind would be if a human being can be described as personally exerting strong nuclear force, just because their subatomic particles do, which I would happily answer "no."
- skepticATX 3y ago
  How about this: understanding is the ability to generalize knowledge and apply it to novel scenarios. This definition is something that humans, and animals for that matter, do every day - both in small and large ways. And this is something that current language models aren't very good at.
  - continuational 3y ago
    What is the test for this? I taught it Firefly, which is an undocumented programming language I'm working on, through conversion. I find it's a lot quicker than any human at picking up syntax and semantics, both in real time and in number of messages, and makes pretty good attempts at writing code in it, as much as you could expect from a human programmer. That is, until you run out of context - is this what you mean?
    - skepticATX 3y ago
      There are plenty of results supporting my assertion; but the tests must be carefully designed. Of course, LLMs are not databases that store exact answers - so it's not enough to ask it something that it hasn't seen, if it's seen something similar (as is likely the case with your programming language). One benchmark that I track closely is ConceptARC, which aims to test generalization and abstraction capabilities. Here is a very recent result that uses the benchmark: https://arxiv.org/abs/2311.09247 https://arxiv.org/abs/2311.09247. Humans correctly solved 91% of the problems, GPT-4 solved 33%, and GPT-4V did much worse than GPT-4.
      - continuational 3y ago
        I wouldn't be surprised if GPT-4 is not too good at visual patterns, given that it's trained on text. Look at the actual prompt in figure 2. I doubt humans would get a 91% score on that.
    - stubybubs 3y ago
      I gave it the three lightbulbs in a closet riddle. https://puzzles.nigelcoldwell.co.uk/seven.htm https://puzzles.nigelcoldwell.co.uk/seven.htm The key complication is "once you've opened the door, you may no longer touch a switch." It gets this. There are many examples of it written out on the web. When I give it a variation and say "you can open the door to look at the bulbs and use the switches all you want" and it is absolutely unable to understand this. To a human it's simple: look at the bulbs and flick the switches. It kept giving me answers about using a special lens to examine the bulbs, using something to detect heat. I explained it in many ways and tried several times. I was paying for GPT-4 at the time as well. I would not consider this thinking. It's unable to make this simple abstraction from its training data. I think 4 looks better than 3 simply because it's got more data, but we're reaching diminishing returns on that, as has been stated.
albertzeyer 3y ago
> You would realize that all we are doing is using statistics to predict what the next item might be. So what does "understanding" really means then? "Understanding" is not really well defined. Either we (humans) do it, but then LLMs might just do it as well, depending on the definition of "understanding", or we both don't do it. But if no-one is really "understanding" anything, then this definition of "understanding" is maybe not useful.
- dboreham 3y ago
  Bingo
- Joeri 3y ago
  Nobody actually understands how humans actually understand something, just like nobody actually understands how LLMs do what they do. Everybody opining about it is doing just that: offering an opinion. Geoff Hinton’s opinion is worth more than someone else’s, but it is still an opinion.
  - im_down_w_otp 3y ago
    I don’t know about the human part, but we absolutely understand how LLMs do what they do. They’re not magic.
    - famouswaffles 3y ago
      No we don't. No it's not "magic". No we don't understand what the black box is doing.
      - brookst 3y ago
        For some values of “we”
        famouswaffles 3y ago
        For every value of we. "I understand the internals of GPT" is the fastest way to demonstrate you have no idea what you're talking about.
satuke 3y ago
Isn't that exactly how humans learn to respond to stimuli? Don't we just try to predict the best next response to everything? Yes, It's statistics but the fun part is nobody is writing this statistical function by hand.
- civilized 3y ago
  LLMs don't have a concept of "best". Only most likely in what they've been trained on. I think LLMs ultimately just take imitation to a creative and sophisticated extreme. And imitation simply doesn't comprise the whole of human intelligence at all, no matter how much it is scaled up. The sophistication of the imitation has some people confused and questioning whether everything can be reduced to imitation. It can't. The ability to imitate seeking a goal isn't identical to the ability to seek a goal. The ability to imitate solving a problem isn't identical to the ability to solve a problem. Imitation is very useful, and the reduction of everything to imitation is an intriguing possibility to consider, but it's ultimately just wrong.
  - lordnacho 3y ago
    Ok, so now we need an example that separates humans from LLMs? I struggle to think of one, maybe someone on HN has a good example. Eg if I'm in middle school and learning quadratic equations, am I imitating solving the problem by plugging in the coefficients? Or am I understanding it? Most of what I see coming out of chatGPT and copilot could be said to be either. If you're generous, it's understanding. If not, it's imitation.
    - civilized 3y ago
      It is very easy to separate humans from LLMs. Humans created math without being given all the answers beforehand. LLMs can't do that yet. When an LLM can create math to solve a problem, we will be much closer to AGI.
      - abm53 3y ago
        You can ask ChatGPT to solve maths problems which are not in its training data, and it will answer an astonishing amount of them correctly. The fact that we have trained it on examples of human-produced maths texts (rather than through interacting with the world over several millennia) seems like more of an implementation detail and not piece of evidence about whether it has “understood” or not.
randyrand 3y ago
the human brain is just a statistical machine.
- brookst 3y ago
  Odds are you’re right
deleted 3y ago
[deleted]
Kiro 3y ago
How do you define understanding when predicting the next color?
karaterobot 3y ago
> You would realize that all we are doing is using statistics to predict what the next item might be. I agree that Hinton's original quote doesn't make sense to me either. I suspect he would leverage the phrase "really good" to explain the difference between ChatGPT and, say, a Markov chain. I think that's a little disingenuous, if that's how he means it, but I don't know if I'm right about that. But I also do not agree that humans use statistics to predict what the next item in a series might be. As evidence, there is the classic example of asking people to predict the next coin toss in the series: "heads, heads, heads, heads, heads, heads, heads...". They'll either guess heads because it's come up so many times already, or because they assume the coin isn't fair, or tails because it's "overdue" to come up, but none of those are based on statistics per se.
troelsSteegin 3y ago
From the article, Gelman's money quote is this: " So I’m not knocking auto-complete; I’m just disagreeing with Hinton’s statement that “by training something to be really good at predicting the next word, you’re actually forcing it to understand.” As a person who does a lot of useful associative reasoning and also a bit of logical understanding, I think they’re different, both in how they feel and also in what they do."
hackerlight 3y ago
Why is "using statistics" mutually exclusive with "understanding"? It would help to carefully define terms. Note that "understanding" doesn't mean it's necessarily conscious. These systems learn high-level representations/abstractions of concepts we humans also use as part of our cognition. The concept of an object, an intuitive physics, the role of specific objects. I don't criticize him for using the word "understanding" to describe this.
porknubbins 3y ago
As a translator I have come to think of it like this- (without the need for defining what “understanding” means) the ability to translate correctly depends on the size of your context. So humans and LLMs are both on the same spectrum with different context sizes of their training set and all facts known in the universe (for a theoretical perfect human) respectively. Doing moderately technical translations, state of the art LLMs may get it about 99% right but you find about 1 word per page where they would have needed to “zoom out” to a wider context to choose the word correctly. Often its an important word that is missed.
usrbinbash 3y ago
> You would realize that all we are doing is using statistics to predict what the next item might be. Two questions: 1. How can humans come up with novel ideas then? Statistically predicting the next item means I am limited to the training set. How do humans formulate new concepts in this scenario? 2. Why can humans learn from their own output then? If all we do is statistically predicting the next token, then, by necessity, feeding our own output back into the model (aka. letting humans learn from it) should degrade human capabilities by overfitting to the training set. But we observe exactly the opposite.
- micromacrofoot 3y ago
  > How can humans come up with novel ideas then? hallucinations, emotions > Why can humans learn from their own output then? if we don't, we suffer
- jameshart 3y ago
  > Statistically predicting the next item means I am limited to the training set Not at all. Extrapolation is perfectly possible in a purely predictive model. It’s one of the things GPTs are best at. In the stream of tokens output by an LLM it’s completely possible for new concepts to emerge, and for it then to continue to use and build on them in the remainder of the stream. You see this simply executed in programming contexts with LLMs where it is able to declare a novel function and then use it.
  - usrbinbash 3y ago
    > with LLMs where it is able to declare a novel function and then use it. Novel as in "implements a new algorithm that has never seen in any form and is actually an improvement over existing methodology"? Here is a little thought experiment: If all the training data in the set says that manned, powered flight is impossible, is a statistical prediction engine trained on that data capable of developing an airplane? In other words, could a very powerful stochastic parrot do what the Wright Brothers did? My best guess is that no, it cannot.
    - 3y ago
canjobear 3y ago
The best “statistic” you could compute to predict the next token would be the meaning of the prompt. A predictor computing meaning would outperform one that is not.
xcv123 3y ago
By "understanding" he means it builds a model of higher order abstractions over the text in order to model the meaning and context of each token in relation to other tokens. This is more sophisticated than a Markov process.
- Jensson 3y ago
  > This is more sophisticated than a Markov process. Nothing prevents a markov process from having a world model. A markov process lacks state and generates a sequence based on statistics and previous entries, so LLMs as they are now are markov processes. Many of the dumb behaviors we see from LLMs today comes from their lack of internal state between tokens, so it don't remember what reason it had for generating the previous token and that means it can easily generate inconsistent answers. So LLMs being a markov process is an important point to highlight since it makes its thinking very different from how humans think.
  - xcv123 3y ago
    The transformer architecture is NOT a Markov process, by mathematical definition of a Markov process. This is not even debatable. It's a mathematical fact. > Many of the dumb behaviors we see from LLMs today comes from their lack of internal state between tokens, so it don't remember what reason it had for generating the previous token and that means it can easily generate inconsistent answers The attention mechanism in the transformer architecture models relations between tokens within the context window, and does the exact opposite of what you are describing here. This is one aspect of LLMs that violates the Markov property.
    - Jensson 3y ago
      > The transformer architecture is NOT a Markov process, by mathematical definition of a Markov process. This is not even debatable. It's a mathematical fact. What, yes it is. > The attention mechanism in the transformer architecture models relations between tokens within the context window, and does the exact opposite of what you are describing here. This is one aspect of LLMs that violates the Markov property. The context window is finite, so that is the previous step. You know the dumb markov chains that are based on bag of words? They also look several words back, they don't just go based on a single word. LLMs are just that but with a way larger lookback and some extra logic there, but none of that changes the fundamental parts to make it no be a markov process. With a large enough context size you could argue it is now fundamentally different in practice, but in theory it is the same. There is no "hidden state", its just the previous n words defines the next word.

continuational 3y ago

I think the most useful definition of understanding something is that you can explain it and use it in context. ChatGPT routinely does both.

jstummbillig 3y ago
And while AI gets better and better and we will remain as touchy as ever about abstract concepts that make us oh so human, how about we say it just can't be understanding, unless a human does, eh, it.
BobaFloutist 3y ago
Someone sufficiently fast and skilled at googling can explain and use in context a lot of things that they don't really properly understand. So unless you're saying that the composite of the googler and of google understand something that neither does individually, your definition has some holes.
- continuational 3y ago
  This is a variation of the Chinese room argument. If you consider understanding an observable property, then the Chinese room in aggregate displays understanding of Chinese. Would you say that humans understand nothing, because atoms don't understand anything, and we're made up of atoms?
  - BobaFloutist 3y ago
    I would say that there is a stronger consensus that a human being can be reasonably described as a single entity than a human being using a reference resource. A more apt comparison to my mind would be if a human being can be described as personally exerting strong nuclear force, just because their subatomic particles do, which I would happily answer "no."
skepticATX 3y ago
How about this: understanding is the ability to generalize knowledge and apply it to novel scenarios. This definition is something that humans, and animals for that matter, do every day - both in small and large ways. And this is something that current language models aren't very good at.
- continuational 3y ago
  What is the test for this? I taught it Firefly, which is an undocumented programming language I'm working on, through conversion. I find it's a lot quicker than any human at picking up syntax and semantics, both in real time and in number of messages, and makes pretty good attempts at writing code in it, as much as you could expect from a human programmer. That is, until you run out of context - is this what you mean?
  - skepticATX 3y ago
    There are plenty of results supporting my assertion; but the tests must be carefully designed. Of course, LLMs are not databases that store exact answers - so it's not enough to ask it something that it hasn't seen, if it's seen something similar (as is likely the case with your programming language). One benchmark that I track closely is ConceptARC, which aims to test generalization and abstraction capabilities. Here is a very recent result that uses the benchmark: https://arxiv.org/abs/2311.09247 https://arxiv.org/abs/2311.09247. Humans correctly solved 91% of the problems, GPT-4 solved 33%, and GPT-4V did much worse than GPT-4.
    - continuational 3y ago
      I wouldn't be surprised if GPT-4 is not too good at visual patterns, given that it's trained on text. Look at the actual prompt in figure 2. I doubt humans would get a 91% score on that.
  - stubybubs 3y ago
    I gave it the three lightbulbs in a closet riddle. https://puzzles.nigelcoldwell.co.uk/seven.htm https://puzzles.nigelcoldwell.co.uk/seven.htm The key complication is "once you've opened the door, you may no longer touch a switch." It gets this. There are many examples of it written out on the web. When I give it a variation and say "you can open the door to look at the bulbs and use the switches all you want" and it is absolutely unable to understand this. To a human it's simple: look at the bulbs and flick the switches. It kept giving me answers about using a special lens to examine the bulbs, using something to detect heat. I explained it in many ways and tried several times. I was paying for GPT-4 at the time as well. I would not consider this thinking. It's unable to make this simple abstraction from its training data. I think 4 looks better than 3 simply because it's got more data, but we're reaching diminishing returns on that, as has been stated.

albertzeyer 3y ago

> You would realize that all we are doing is using statistics to predict what the next item might be. So what does "understanding" really means then? "Understanding" is not really well defined. Either we (humans) do it, but then LLMs might just do it as well, depending on the definition of "understanding", or we both don't do it. But if no-one is really "understanding" anything, then this definition of "understanding" is maybe not useful.

dboreham 3y ago
Bingo
Joeri 3y ago
Nobody actually understands how humans actually understand something, just like nobody actually understands how LLMs do what they do. Everybody opining about it is doing just that: offering an opinion. Geoff Hinton’s opinion is worth more than someone else’s, but it is still an opinion.
- im_down_w_otp 3y ago
  I don’t know about the human part, but we absolutely understand how LLMs do what they do. They’re not magic.
  - famouswaffles 3y ago
    No we don't. No it's not "magic". No we don't understand what the black box is doing.
    - brookst 3y ago
      For some values of “we”
      - famouswaffles 3y ago
        For every value of we. "I understand the internals of GPT" is the fastest way to demonstrate you have no idea what you're talking about.

satuke 3y ago

Isn't that exactly how humans learn to respond to stimuli? Don't we just try to predict the best next response to everything? Yes, It's statistics but the fun part is nobody is writing this statistical function by hand.

civilized 3y ago
LLMs don't have a concept of "best". Only most likely in what they've been trained on. I think LLMs ultimately just take imitation to a creative and sophisticated extreme. And imitation simply doesn't comprise the whole of human intelligence at all, no matter how much it is scaled up. The sophistication of the imitation has some people confused and questioning whether everything can be reduced to imitation. It can't. The ability to imitate seeking a goal isn't identical to the ability to seek a goal. The ability to imitate solving a problem isn't identical to the ability to solve a problem. Imitation is very useful, and the reduction of everything to imitation is an intriguing possibility to consider, but it's ultimately just wrong.
- lordnacho 3y ago
  Ok, so now we need an example that separates humans from LLMs? I struggle to think of one, maybe someone on HN has a good example. Eg if I'm in middle school and learning quadratic equations, am I imitating solving the problem by plugging in the coefficients? Or am I understanding it? Most of what I see coming out of chatGPT and copilot could be said to be either. If you're generous, it's understanding. If not, it's imitation.
  - civilized 3y ago
    It is very easy to separate humans from LLMs. Humans created math without being given all the answers beforehand. LLMs can't do that yet. When an LLM can create math to solve a problem, we will be much closer to AGI.
    - abm53 3y ago
      You can ask ChatGPT to solve maths problems which are not in its training data, and it will answer an astonishing amount of them correctly. The fact that we have trained it on examples of human-produced maths texts (rather than through interacting with the world over several millennia) seems like more of an implementation detail and not piece of evidence about whether it has “understood” or not.

randyrand 3y ago

the human brain is just a statistical machine.

brookst 3y ago
Odds are you’re right

deleted 3y ago

[deleted]

Kiro 3y ago

How do you define understanding when predicting the next color?

karaterobot 3y ago

> You would realize that all we are doing is using statistics to predict what the next item might be. I agree that Hinton's original quote doesn't make sense to me either. I suspect he would leverage the phrase "really good" to explain the difference between ChatGPT and, say, a Markov chain. I think that's a little disingenuous, if that's how he means it, but I don't know if I'm right about that. But I also do not agree that humans use statistics to predict what the next item in a series might be. As evidence, there is the classic example of asking people to predict the next coin toss in the series: "heads, heads, heads, heads, heads, heads, heads...". They'll either guess heads because it's come up so many times already, or because they assume the coin isn't fair, or tails because it's "overdue" to come up, but none of those are based on statistics per se.

troelsSteegin 3y ago

From the article, Gelman's money quote is this: " So I’m not knocking auto-complete; I’m just disagreeing with Hinton’s statement that “by training something to be really good at predicting the next word, you’re actually forcing it to understand.” As a person who does a lot of useful associative reasoning and also a bit of logical understanding, I think they’re different, both in how they feel and also in what they do."

hackerlight 3y ago

Why is "using statistics" mutually exclusive with "understanding"? It would help to carefully define terms. Note that "understanding" doesn't mean it's necessarily conscious. These systems learn high-level representations/abstractions of concepts we humans also use as part of our cognition. The concept of an object, an intuitive physics, the role of specific objects. I don't criticize him for using the word "understanding" to describe this.

porknubbins 3y ago

As a translator I have come to think of it like this- (without the need for defining what “understanding” means) the ability to translate correctly depends on the size of your context. So humans and LLMs are both on the same spectrum with different context sizes of their training set and all facts known in the universe (for a theoretical perfect human) respectively. Doing moderately technical translations, state of the art LLMs may get it about 99% right but you find about 1 word per page where they would have needed to “zoom out” to a wider context to choose the word correctly. Often its an important word that is missed.

usrbinbash 3y ago

> You would realize that all we are doing is using statistics to predict what the next item might be. Two questions: 1. How can humans come up with novel ideas then? Statistically predicting the next item means I am limited to the training set. How do humans formulate new concepts in this scenario? 2. Why can humans learn from their own output then? If all we do is statistically predicting the next token, then, by necessity, feeding our own output back into the model (aka. letting humans learn from it) should degrade human capabilities by overfitting to the training set. But we observe exactly the opposite.

micromacrofoot 3y ago
> How can humans come up with novel ideas then? hallucinations, emotions > Why can humans learn from their own output then? if we don't, we suffer
jameshart 3y ago
> Statistically predicting the next item means I am limited to the training set Not at all. Extrapolation is perfectly possible in a purely predictive model. It’s one of the things GPTs are best at. In the stream of tokens output by an LLM it’s completely possible for new concepts to emerge, and for it then to continue to use and build on them in the remainder of the stream. You see this simply executed in programming contexts with LLMs where it is able to declare a novel function and then use it.
- usrbinbash 3y ago
  > with LLMs where it is able to declare a novel function and then use it. Novel as in "implements a new algorithm that has never seen in any form and is actually an improvement over existing methodology"? Here is a little thought experiment: If all the training data in the set says that manned, powered flight is impossible, is a statistical prediction engine trained on that data capable of developing an airplane? In other words, could a very powerful stochastic parrot do what the Wright Brothers did? My best guess is that no, it cannot.
  - 3y ago

canjobear 3y ago

The best “statistic” you could compute to predict the next token would be the meaning of the prompt. A predictor computing meaning would outperform one that is not.

xcv123 3y ago

By "understanding" he means it builds a model of higher order abstractions over the text in order to model the meaning and context of each token in relation to other tokens. This is more sophisticated than a Markov process.

Jensson 3y ago
> This is more sophisticated than a Markov process. Nothing prevents a markov process from having a world model. A markov process lacks state and generates a sequence based on statistics and previous entries, so LLMs as they are now are markov processes. Many of the dumb behaviors we see from LLMs today comes from their lack of internal state between tokens, so it don't remember what reason it had for generating the previous token and that means it can easily generate inconsistent answers. So LLMs being a markov process is an important point to highlight since it makes its thinking very different from how humans think.
- xcv123 3y ago
  The transformer architecture is NOT a Markov process, by mathematical definition of a Markov process. This is not even debatable. It's a mathematical fact. > Many of the dumb behaviors we see from LLMs today comes from their lack of internal state between tokens, so it don't remember what reason it had for generating the previous token and that means it can easily generate inconsistent answers The attention mechanism in the transformer architecture models relations between tokens within the context window, and does the exact opposite of what you are describing here. This is one aspect of LLMs that violates the Markov property.
  - Jensson 3y ago
    > The transformer architecture is NOT a Markov process, by mathematical definition of a Markov process. This is not even debatable. It's a mathematical fact. What, yes it is. > The attention mechanism in the transformer architecture models relations between tokens within the context window, and does the exact opposite of what you are describing here. This is one aspect of LLMs that violates the Markov property. The context window is finite, so that is the previous step. You know the dumb markov chains that are based on bag of words? They also look several words back, they don't just go based on a single word. LLMs are just that but with a way larger lookback and some extra logic there, but none of that changes the fundamental parts to make it no be a markov process. With a large enough context size you could argue it is now fundamentally different in practice, but in theory it is the same. There is no "hidden state", its just the previous n words defines the next word.