27 ms·

> Memory is not understanding, even though they often look similar. I’m curious about this because I used to think I knew what “understanding” is, but now I do

> Memory is not understanding, even though they often look similar. I’m curious about this because I used to think I knew what “understanding” is, but now I don’t think I do. A lot of what we think we “understand” is actually memorized. And with sufficient information, memory is indistinguishable from understanding. A computer can memorize all of the best tic-tac-toe moves. This memorization is indistinguishable from understanding. In fact, it’s better because it’s cached. Empirically, understanding can be tested with a novel set of challenge questions. If no such question set can be produced (i.e. the entire domain has been memorized) or the subject answers the question set with no apparent repeatable flaws, we must admit the subject understands the domain. Anything further, any attempt to peer inside the box (other than to identify repeatable flaws), is not a productive use of time. Further, we should keep in mind that LLMs aren’t usually actually regurgitating training data verbatim. Their parameters store a statistical representation of the training data and are orders of magnitude smaller than a lossless compressed version of that data. In this way, it is analogous to MCTS analysis of a chess position: rollouts can give very deep information even if that “understanding” is quite alien to us.

deleted 3y ago
[deleted]
axus 3y ago
I'd say "know" is what we've memorized, "understand" is what we can support an argument for... though of course you can memorize the argument and not understand the ideas comprising the argument !
- LeonenTheDK 3y ago
  It's t̶u̶r̶t̶l̶e̶s̶ understanding all the way down.
godelski 3y ago
> A lot of what we think we “understand” is actually memorized. For sure. But these definitions are not well defined, despite thousands of years of research. One point I think everyone in that chain would agree upon is that with respect to understanding, a thing would not explain its answer by giving evidence to the contrary. If someone did so you rightfully would say "You don't know [understand] what you're talking about." > Empirically, understanding can be tested with a novel set of challenge questions. I have to stop you right there. Empirical measures are always signals, not proofs. This too has been long established for hundreds of years and is the reason many engineers and experimental physicists fight. Why experimental physicists and theoretical physicists fight. Empirical evidence is always limited to its context window and we have to be VERY clear at what those limitations are. This is the lesson of Goodhart's Law, not that people will exploit the measure (though that is also important). Every intro ML course shows RL agents metric hacking, and this is always a result of a limited context. So I want to be clear, that empirical data doesn't test a hypothesis but rather tests the null-hypothesis. That's important because the set of possible hypotheses is just decreased through empirical testing, rather than reducing the set of possible results a single element set. It rejects hypotheses, not proves them. We must also understand that testing understanding is similarly an unsolved problem. One too that has been questioned for millennia. > Anything further, any attempt to peer inside the box (other than to identify repeatable flaws), is not a productive use of time. This is strange to me. The parenthetical statement is unbounded and justifies what you are arguing against. Second, just because we can't prove a specific hypothesis (in most cases) doesn't mean it isn't useful. Rejection is how the vast majority of science works and this has been highly successful. Limiting potential is clearly a helpful tool as it increases your odds of a correct answer (which is why it is a common strategy for multiple choice testing). > Further, we should keep in mind that LLMs aren’t usually actually regurgitating training data verbatim. Yes, this is the stochastic part of the term stochastic parrot. No reasonable researcher is suggesting that LLMs only recite. Every one of us recognizes that they can generate things that were not handed to it. In fact, a hallucination is an explicit example of this. The failure to respond to my answer correctly is an example of it generating new data. (It is also an example of it following a statistical pattern). But we do also need to be careful about our distinctions of generalization vs overfitting. There are clearly certain areas that are overfit (as I demonstrated). This is extra difficult in models that have been trained on datasets which you are not privy to. But we can also see good examples of how the code LLMs are overfit. I have written a number of comments on this site about exactly this, which you're welcome to search my history for (see "HumanEval" with my name). Here's one such comment id=35806152
namaria 3y ago
I think ChatGPT makes quite clear that understanding language and understanding underlying concepts expressed in language are different things. While it seems to be competent in processing and producing language, it appears to have no underlying concept of logic, physics, society, etc. We shouldn't conflate different kinds of understanding. There's no need to say "it's either shuffling tokens blindly or showing intelligence".
- janalsncm 3y ago
  > it appears to have no underlying concept of logic, physics, society, etc. Sure. ChatGPT is quite limited and gives incorrect and contradictory responses at times. And most importantly (imo) it is unable to update an interpretable knowledge base as real life facts change. But the fact that a particular instance of LLM has this unwanted behavior is not categorical evidence that all LLMs will show the behavior. GPT2 is pretty pathetic compared to GPT3, and ChatGPT is even more impressive. I’m agnostic as to whether statistical language models can ever overcome their current limitations, but given previous emergent behavior I wouldn’t rule it out. GPT4’s context window is something like 32k tokens. Someday that may be orders of magnitude larger, and it may be possible to fit an up to date copy of all of Wikipedia inside (or a condensed version), as well as the current conversation. It might seem crazy but it’s foreseeable.
  - namaria 3y ago
    > given previous emergent behavior I wouldn’t rule it out. I find this an interesting turn of phrase, because "emergent behavior" is often used by those holding a reductionist view to hand wave away complexity that cannot be reduced to the analyzed elements. In a way it makes me think of Greek sophists, who would cogently argue for two opposing view points and making a convincing case for both. It's way easier to make a verbal case about something than to prove it true, or false.
- perfmode 3y ago
  language is just concepts. understanding language and understanding concepts is the same thing. to think one can understand language and not understand concepts is to not understand language.

deleted 3y ago

[deleted]

axus 3y ago

I'd say "know" is what we've memorized, "understand" is what we can support an argument for... though of course you can memorize the argument and not understand the ideas comprising the argument !

LeonenTheDK 3y ago
It's t̶u̶r̶t̶l̶e̶s̶ understanding all the way down.

godelski 3y ago

> A lot of what we think we “understand” is actually memorized. For sure. But these definitions are not well defined, despite thousands of years of research. One point I think everyone in that chain would agree upon is that with respect to understanding, a thing would not explain its answer by giving evidence to the contrary. If someone did so you rightfully would say "You don't know [understand] what you're talking about." > Empirically, understanding can be tested with a novel set of challenge questions. I have to stop you right there. Empirical measures are always signals, not proofs. This too has been long established for hundreds of years and is the reason many engineers and experimental physicists fight. Why experimental physicists and theoretical physicists fight. Empirical evidence is always limited to its context window and we have to be VERY clear at what those limitations are. This is the lesson of Goodhart's Law, not that people will exploit the measure (though that is also important). Every intro ML course shows RL agents metric hacking, and this is always a result of a limited context. So I want to be clear, that empirical data doesn't test a hypothesis but rather tests the null-hypothesis. That's important because the set of possible hypotheses is just decreased through empirical testing, rather than reducing the set of possible results a single element set. It rejects hypotheses, not proves them. We must also understand that testing understanding is similarly an unsolved problem. One too that has been questioned for millennia. > Anything further, any attempt to peer inside the box (other than to identify repeatable flaws), is not a productive use of time. This is strange to me. The parenthetical statement is unbounded and justifies what you are arguing against. Second, just because we can't prove a specific hypothesis (in most cases) doesn't mean it isn't useful. Rejection is how the vast majority of science works and this has been highly successful. Limiting potential is clearly a helpful tool as it increases your odds of a correct answer (which is why it is a common strategy for multiple choice testing). > Further, we should keep in mind that LLMs aren’t usually actually regurgitating training data verbatim. Yes, this is the stochastic part of the term stochastic parrot. No reasonable researcher is suggesting that LLMs only recite. Every one of us recognizes that they can generate things that were not handed to it. In fact, a hallucination is an explicit example of this. The failure to respond to my answer correctly is an example of it generating new data. (It is also an example of it following a statistical pattern). But we do also need to be careful about our distinctions of generalization vs overfitting. There are clearly certain areas that are overfit (as I demonstrated). This is extra difficult in models that have been trained on datasets which you are not privy to. But we can also see good examples of how the code LLMs are overfit. I have written a number of comments on this site about exactly this, which you're welcome to search my history for (see "HumanEval" with my name). Here's one such comment id=35806152

namaria 3y ago

I think ChatGPT makes quite clear that understanding language and understanding underlying concepts expressed in language are different things. While it seems to be competent in processing and producing language, it appears to have no underlying concept of logic, physics, society, etc. We shouldn't conflate different kinds of understanding. There's no need to say "it's either shuffling tokens blindly or showing intelligence".

janalsncm 3y ago
> it appears to have no underlying concept of logic, physics, society, etc. Sure. ChatGPT is quite limited and gives incorrect and contradictory responses at times. And most importantly (imo) it is unable to update an interpretable knowledge base as real life facts change. But the fact that a particular instance of LLM has this unwanted behavior is not categorical evidence that all LLMs will show the behavior. GPT2 is pretty pathetic compared to GPT3, and ChatGPT is even more impressive. I’m agnostic as to whether statistical language models can ever overcome their current limitations, but given previous emergent behavior I wouldn’t rule it out. GPT4’s context window is something like 32k tokens. Someday that may be orders of magnitude larger, and it may be possible to fit an up to date copy of all of Wikipedia inside (or a condensed version), as well as the current conversation. It might seem crazy but it’s foreseeable.
- namaria 3y ago
  > given previous emergent behavior I wouldn’t rule it out. I find this an interesting turn of phrase, because "emergent behavior" is often used by those holding a reductionist view to hand wave away complexity that cannot be reduced to the analyzed elements. In a way it makes me think of Greek sophists, who would cogently argue for two opposing view points and making a convincing case for both. It's way easier to make a verbal case about something than to prove it true, or false.
perfmode 3y ago
language is just concepts. understanding language and understanding concepts is the same thing. to think one can understand language and not understand concepts is to not understand language.