14 ms·

This is a fair question: LLMs do challenge the easy assumption (as made, for example, in Searle's "Chinese Room" thought experiment) that computers cannot possi

This is a fair question: LLMs do challenge the easy assumption (as made, for example, in Searle's "Chinese Room" thought experiment) that computers cannot possibly understand things. Here, however, I would say that if an LLM can be said to have understanding or knowledge of something, it is of the patterns of token occurrences to be found in the use of language. It is not clear that this also grants the LLM any understanding that this language refers to an external world which operates in response to causes which are independent of what is or might be said about it.

ninetyninenine 3y ago
Explain sora. It must have of course a blurry understanding of reality to even produce those videos. I think we are way past the point of debate here. LLMs are not stochastic parrots. LLMs do understand an aspect of reality. Even the LLMs that are weaker than sora understand things. What is debatable is whether LLMs are conscious. But whether it can understand something is a pretty clear yes. But does it understand everything? No.
- solarhexes 3y ago
  If by “understand” you mean “can model reasonably accurately much of the time” then maybe you’ll find consensus. But that’s not a universal definition of “understand”. For example, if I asked you whether you “understand” ballistic flight, and you produced a table that you interpolate from instead of a quadratic, then I would not feel that you understand it, even though you can kinda sorta model it. And even if you do, if you didn’t produce the universal gravitation formula, I would still wonder how “deeply” you understand. So it’s not like “understand” is a binary I suppose.
  - eszed 3y ago
    I think there are two axes: reason about and intuit. I "understand" ballistic flight when I can calculate a solution that puts an artillery round on target. I also "understand" ballistic flight when I make a free throw with a basketball. On writing that, I have an instinct to revise it to move the locus of understanding in the first example to the people who calculated the ballistic tables, based on physics first-principles. That would be more accurate, but my mistake highlights something interesting: an artillery officer / spotter simultaneously uses both. Is theirs a "deeper" / "truer" understanding? I don't think it is. I don't know what I think that means, for humans or AI.
  - ninetyninenine 3y ago
    Well what would you need to see to prove understanding? That's the metric here. Both the LLM and the human brain are black boxes. But we claim the human brain understands things while the LLM does not. Thus what output would you expect for either of these boxes to demonstrate true understanding to your question?
    - mannykannot 3y ago
      It is interesting that you are demanding a metric here, as yours appears to be like duck typing: in effect, if it quacks like a human... Defining "understanding" is difficult (epistemology struggles with the apparently simpler task of defining knowledge), but if I saw a dialogue between two LLMs figuring out something about the external world that they did not initially have much to say about, I would find that pretty convincing.
      - ninetyninenine 3y ago
        Without a metric no position can be made. All conversation about this topic is just conjecture with no path to a conclusion.
        mannykannot 3y ago
        This is a common misunderstanding, one also seen with regard to definitions. When applied to knowledge acquisition, it suffers from a fairly obvious bootstrapping problem, which goes away when you realize that metrics and definitions are rewritten and refined as our knowledge increases. Just look at what has happened to concepts of matter and energy over the last century or so. You are free to disagree with this, but I feel your metric for understanding resembles the Turing test, while the sort of thing I have proposed here, which involves AIs interacting with each other, is a refinement that makes a step away from defining understanding and intelligence as being just whatever human judges recognize as such (it still depends on human judgement, but I think one could analyze the sort of dialogue I am envisioning more objectively than in a Turing test.)
        3y ago
  - cultureswitch 3y ago
    Are you telling me that WW1 artillery crews didn't understand ballistics? Because they were using tables. There's no difference between doing something that works without understanding and doing the exact same thing with understanding.
    - solarhexes 3y ago
      You’ve decided that your definition of “understanding” is correct. Ok.
      - mannykannot 3y ago
        The author of the post to which you are replying seems to be defining "understanding" as merely meaning "able to do something."
        ninetyninenine 3y ago
        The author of the post is saying that understanding something can't be defined because we can't even know how the human brain works. It is a black box. The author is saying at best you can only set benchmark comparisons. We just assume all humans have the capability of understanding without even really defining the meaning of understanding. And if a machine can mimic human behavior to it must also understand. That is literally how far we can go from a logical standpoint. It's the furthest we can go in terms of classifying things as either capable of understanding or not capable or close. What you're not seeing is the LLM is not only mimicking human output to a high degree. It can even produce output that is superior to what humans can produce.
        mannykannot 3y ago
        What the author of the post actually said - and I am quoting, to make it clear that I'm not putting my spin on someone else's opinion - was "There's no difference between doing something that works without understanding and doing the exact same thing with understanding."
- andsoitis 3y ago
  > I think we are way past the point of debate here. LLMs are not stochastic parrots. LLMs do understand an aspect of reality. Even the LLMs that are weaker than sora understand things. What is one such aspect? (I'm not asking in order to debate it here, but more because I want to test / research it on my own time)
  - ninetyninenine 3y ago
    I pay for chatGPT so it depends on if you pay for that or not. I think it's worth it because whether it understands things or not chatGPT represents a paradigm shift in human history. You'll need it because it's currently the best conversational LLM out there and the one that shows the most compelling evidence. Basically you just spend a lot of time with chatGPT4 and ask it deep questions that don't exist in it's dataset. get creative. The LLM will output answers that demonstrate a lack of understanding, but it will also demonstrate answers that display a remarkable amount of understanding. Both sets of answers exist and people often cite the wrong answers as evidence for lack of understanding but they're setting bar too high. The fact that many of these answers do demonstrate understanding of concepts makes it very very compelling. Take for example Rock Paper Scissors. https://chat.openai.com/share/ca22397c-2950-4919-bb79-6def64377512 https://chat.openai.com/share/ca22397c-2950-4919-bb79-6def64... This entire conversation thread I believe does not exist in a parallel form in it's data set. It demonstrates understanding of RPS beyond the confines of text, it demonstrates understanding of simultaneity EVEN when the LLM wholly lives in a world of turn based questions and responses, it understands itself relative to simultaneity, it tries to find solutions around it's own problem, it understands how to use creativity and solutions such as cryptography to solve the problem of RPS when playing with it, it also understands the weaknesses of it's own solutions. Conversations such as this show that chatGPT displays remarkable understanding of the world. There are conversations that are opposite to this that demonstrate LLMs displaying an obvious lack of understanding. But the existence of these conversation that lack understanding does NOT negate the ones that do demonstrate understanding. The fact that partial understanding even exists is a milestone for AI. This isn't Anthropomorphism. People are throwing this word trying to get people to recognize their own biases without realizing that it's just demonstrating their own biases. We literally can't even define "understanding" and both LLMs and the human brain are black boxes. Making adamant claims saying that LLMs don't understand anything without addressing this fact is itself a form of bias. The way I address the problem above is that I just define a bar. I define humans as the bar of "understanding" without defining what understanding means itself. Then if any machine begins approaching this bar in terms of input and output matching human responses, then this is logically indistinguishable from approaching "understanding". That's literally the best metric we have.
- beardedwizard 3y ago
  I do not understand these comments at all. Sora was trained on billions of frames from video and images - they were tagged with words like "ballistic missile launch" and "cinematic shot" and it simply predicts the pixels like every other model. It stores what we showed it, and reproduces it when we ask - this has nothing to do with understanding and everything to do with parroting. The fact that it's now a stream of images instead of just 1 changes nothing about it.
  - ninetyninenine 3y ago
    What is the difference between a machine that for all intents and purposes appears to understand something to a degree of 100 percent versus a human? Both the machine and the human are a black box. The human brain is not completely understood and the LLM is only trivially understood at a high level through the lens of stochastic curve fitting. When something produces output that imitates the output related to a human that we claim "understands" things that is objectively understanding because we cannot penetrate the black box of human intelligence or machine intelligence to determine further. In fact in terms of image generation the LLM is superior. It will generate video output superior to what a human can generate. Now mind you the human brain has a classifier and can identify flaws but try watching a human with Photoshop to try to even draw one frame of those videos.. it will be horrible. Does this indicate that humans lack understanding? Again, hard to answer because we are dealing with black boxes so it's hard to pinpoint what understanding something even means. We can however set a bar. A metric. And we can define that bar as humans. all humans understand things. Any machine that approaches human input and output capabilities is approaching human understanding.
    - Jensson 3y ago
      > What is the difference between a machine that for all intents and purposes appears to understand something to a degree of 100 percent versus a human? There is no such difference, we evaluate that based on their output. We see these massive model make silly errors that nobody who understands it would make, thus we say the model doesn't understand. We do that for humans as well. For example, for Sora in the video with the dog in the windos, we see the dog walk straight through the window shutters, so Sora doesn't understand physics or depth. We also see it drawing the dogs shadow on the wall very thin, much smaller than the dog itself, it obviously drew that shadow as if it was cast on the ground and not a wall, it would have been very large shadow on that wall. The shadows from the shutters were normal, because Sora are used to those shadows being on a wall. Hence we can say Sora doesn't understand physics or shadows, but it has very impressive heuristics about those, the dog accurately places its paws on the platforms etc, and the paws shadows were right. But we know those were just basic heuristics since the dog walked through the shutters and its body cast shadow in the wrong way meaning Sora only handles very common cases and fails as soon as things are in an unexpected envionment.
- mannykannot 3y ago
  I fail to see how changing the output medium from sentences to movie frames is a difference that I need to account for - the principle is the same either way. I feel you are missing an important part of my point here. I am not taking a position on whether LLMs can be said to understand anything at all; I am saying that I seriously doubt that LLMs understand that the language they receive refers to an external world.
cultureswitch 3y ago
Should it matter how the object of debate interacts and probes the external world? We sense the world through specialized cells connected to neurons. There's nothing to prevent LLMs doing functionally the same thing. Both human brains and LLMs have information inputs and outputs, there's nothing that can go through one which can't go through the other.
- mannykannot 3y ago
  A current LLM does not interact with the external world in a way that would seem to lead to an understanding of it. It emits a response to a prompt, and then reverts to passively waiting for the next one. There's no way for it to anticipate something will happen in response, and thereby get the feedback needed to realize that there is more to the language it receives than is contained in the statistical relationships between its tokens. If its model is updated in the interim, it is unaware, afterwards, that a change has occurred.

ninetyninenine 3y ago

Explain sora. It must have of course a blurry understanding of reality to even produce those videos. I think we are way past the point of debate here. LLMs are not stochastic parrots. LLMs do understand an aspect of reality. Even the LLMs that are weaker than sora understand things. What is debatable is whether LLMs are conscious. But whether it can understand something is a pretty clear yes. But does it understand everything? No.

solarhexes 3y ago
If by “understand” you mean “can model reasonably accurately much of the time” then maybe you’ll find consensus. But that’s not a universal definition of “understand”. For example, if I asked you whether you “understand” ballistic flight, and you produced a table that you interpolate from instead of a quadratic, then I would not feel that you understand it, even though you can kinda sorta model it. And even if you do, if you didn’t produce the universal gravitation formula, I would still wonder how “deeply” you understand. So it’s not like “understand” is a binary I suppose.
- eszed 3y ago
  I think there are two axes: reason about and intuit. I "understand" ballistic flight when I can calculate a solution that puts an artillery round on target. I also "understand" ballistic flight when I make a free throw with a basketball. On writing that, I have an instinct to revise it to move the locus of understanding in the first example to the people who calculated the ballistic tables, based on physics first-principles. That would be more accurate, but my mistake highlights something interesting: an artillery officer / spotter simultaneously uses both. Is theirs a "deeper" / "truer" understanding? I don't think it is. I don't know what I think that means, for humans or AI.
- ninetyninenine 3y ago
  Well what would you need to see to prove understanding? That's the metric here. Both the LLM and the human brain are black boxes. But we claim the human brain understands things while the LLM does not. Thus what output would you expect for either of these boxes to demonstrate true understanding to your question?
  - mannykannot 3y ago
    It is interesting that you are demanding a metric here, as yours appears to be like duck typing: in effect, if it quacks like a human... Defining "understanding" is difficult (epistemology struggles with the apparently simpler task of defining knowledge), but if I saw a dialogue between two LLMs figuring out something about the external world that they did not initially have much to say about, I would find that pretty convincing.
    - ninetyninenine 3y ago
      Without a metric no position can be made. All conversation about this topic is just conjecture with no path to a conclusion.
      - mannykannot 3y ago
        This is a common misunderstanding, one also seen with regard to definitions. When applied to knowledge acquisition, it suffers from a fairly obvious bootstrapping problem, which goes away when you realize that metrics and definitions are rewritten and refined as our knowledge increases. Just look at what has happened to concepts of matter and energy over the last century or so. You are free to disagree with this, but I feel your metric for understanding resembles the Turing test, while the sort of thing I have proposed here, which involves AIs interacting with each other, is a refinement that makes a step away from defining understanding and intelligence as being just whatever human judges recognize as such (it still depends on human judgement, but I think one could analyze the sort of dialogue I am envisioning more objectively than in a Turing test.)
        3y ago
- cultureswitch 3y ago
  Are you telling me that WW1 artillery crews didn't understand ballistics? Because they were using tables. There's no difference between doing something that works without understanding and doing the exact same thing with understanding.
  - solarhexes 3y ago
    You’ve decided that your definition of “understanding” is correct. Ok.
    - mannykannot 3y ago
      The author of the post to which you are replying seems to be defining "understanding" as merely meaning "able to do something."
      - ninetyninenine 3y ago
        The author of the post is saying that understanding something can't be defined because we can't even know how the human brain works. It is a black box. The author is saying at best you can only set benchmark comparisons. We just assume all humans have the capability of understanding without even really defining the meaning of understanding. And if a machine can mimic human behavior to it must also understand. That is literally how far we can go from a logical standpoint. It's the furthest we can go in terms of classifying things as either capable of understanding or not capable or close. What you're not seeing is the LLM is not only mimicking human output to a high degree. It can even produce output that is superior to what humans can produce.
        mannykannot 3y ago
        What the author of the post actually said - and I am quoting, to make it clear that I'm not putting my spin on someone else's opinion - was "There's no difference between doing something that works without understanding and doing the exact same thing with understanding."
andsoitis 3y ago
> I think we are way past the point of debate here. LLMs are not stochastic parrots. LLMs do understand an aspect of reality. Even the LLMs that are weaker than sora understand things. What is one such aspect? (I'm not asking in order to debate it here, but more because I want to test / research it on my own time)
- ninetyninenine 3y ago
  I pay for chatGPT so it depends on if you pay for that or not. I think it's worth it because whether it understands things or not chatGPT represents a paradigm shift in human history. You'll need it because it's currently the best conversational LLM out there and the one that shows the most compelling evidence. Basically you just spend a lot of time with chatGPT4 and ask it deep questions that don't exist in it's dataset. get creative. The LLM will output answers that demonstrate a lack of understanding, but it will also demonstrate answers that display a remarkable amount of understanding. Both sets of answers exist and people often cite the wrong answers as evidence for lack of understanding but they're setting bar too high. The fact that many of these answers do demonstrate understanding of concepts makes it very very compelling. Take for example Rock Paper Scissors. https://chat.openai.com/share/ca22397c-2950-4919-bb79-6def64377512 https://chat.openai.com/share/ca22397c-2950-4919-bb79-6def64... This entire conversation thread I believe does not exist in a parallel form in it's data set. It demonstrates understanding of RPS beyond the confines of text, it demonstrates understanding of simultaneity EVEN when the LLM wholly lives in a world of turn based questions and responses, it understands itself relative to simultaneity, it tries to find solutions around it's own problem, it understands how to use creativity and solutions such as cryptography to solve the problem of RPS when playing with it, it also understands the weaknesses of it's own solutions. Conversations such as this show that chatGPT displays remarkable understanding of the world. There are conversations that are opposite to this that demonstrate LLMs displaying an obvious lack of understanding. But the existence of these conversation that lack understanding does NOT negate the ones that do demonstrate understanding. The fact that partial understanding even exists is a milestone for AI. This isn't Anthropomorphism. People are throwing this word trying to get people to recognize their own biases without realizing that it's just demonstrating their own biases. We literally can't even define "understanding" and both LLMs and the human brain are black boxes. Making adamant claims saying that LLMs don't understand anything without addressing this fact is itself a form of bias. The way I address the problem above is that I just define a bar. I define humans as the bar of "understanding" without defining what understanding means itself. Then if any machine begins approaching this bar in terms of input and output matching human responses, then this is logically indistinguishable from approaching "understanding". That's literally the best metric we have.
beardedwizard 3y ago
I do not understand these comments at all. Sora was trained on billions of frames from video and images - they were tagged with words like "ballistic missile launch" and "cinematic shot" and it simply predicts the pixels like every other model. It stores what we showed it, and reproduces it when we ask - this has nothing to do with understanding and everything to do with parroting. The fact that it's now a stream of images instead of just 1 changes nothing about it.
- ninetyninenine 3y ago
  What is the difference between a machine that for all intents and purposes appears to understand something to a degree of 100 percent versus a human? Both the machine and the human are a black box. The human brain is not completely understood and the LLM is only trivially understood at a high level through the lens of stochastic curve fitting. When something produces output that imitates the output related to a human that we claim "understands" things that is objectively understanding because we cannot penetrate the black box of human intelligence or machine intelligence to determine further. In fact in terms of image generation the LLM is superior. It will generate video output superior to what a human can generate. Now mind you the human brain has a classifier and can identify flaws but try watching a human with Photoshop to try to even draw one frame of those videos.. it will be horrible. Does this indicate that humans lack understanding? Again, hard to answer because we are dealing with black boxes so it's hard to pinpoint what understanding something even means. We can however set a bar. A metric. And we can define that bar as humans. all humans understand things. Any machine that approaches human input and output capabilities is approaching human understanding.
  - Jensson 3y ago
    > What is the difference between a machine that for all intents and purposes appears to understand something to a degree of 100 percent versus a human? There is no such difference, we evaluate that based on their output. We see these massive model make silly errors that nobody who understands it would make, thus we say the model doesn't understand. We do that for humans as well. For example, for Sora in the video with the dog in the windos, we see the dog walk straight through the window shutters, so Sora doesn't understand physics or depth. We also see it drawing the dogs shadow on the wall very thin, much smaller than the dog itself, it obviously drew that shadow as if it was cast on the ground and not a wall, it would have been very large shadow on that wall. The shadows from the shutters were normal, because Sora are used to those shadows being on a wall. Hence we can say Sora doesn't understand physics or shadows, but it has very impressive heuristics about those, the dog accurately places its paws on the platforms etc, and the paws shadows were right. But we know those were just basic heuristics since the dog walked through the shutters and its body cast shadow in the wrong way meaning Sora only handles very common cases and fails as soon as things are in an unexpected envionment.
mannykannot 3y ago
I fail to see how changing the output medium from sentences to movie frames is a difference that I need to account for - the principle is the same either way. I feel you are missing an important part of my point here. I am not taking a position on whether LLMs can be said to understand anything at all; I am saying that I seriously doubt that LLMs understand that the language they receive refers to an external world.

cultureswitch 3y ago

Should it matter how the object of debate interacts and probes the external world? We sense the world through specialized cells connected to neurons. There's nothing to prevent LLMs doing functionally the same thing. Both human brains and LLMs have information inputs and outputs, there's nothing that can go through one which can't go through the other.

mannykannot 3y ago
A current LLM does not interact with the external world in a way that would seem to lead to an understanding of it. It emits a response to a prompt, and then reverts to passively waiting for the next one. There's no way for it to anticipate something will happen in response, and thereby get the feedback needed to realize that there is more to the language it receives than is contained in the statistical relationships between its tokens. If its model is updated in the interim, it is unaware, afterwards, that a change has occurred.