4 ms·

But LLMs use and reason with the data, so the word “know” is probably appropriate. Think of it as a analogy, if you want, but it’s reasonable to draw a comparis

But LLMs use and reason with the data, so the word “know” is probably appropriate. Think of it as a analogy, if you want, but it’s reasonable to draw a comparison with what humans do. Even without the reasoning part I’m sure people already use the word “know” to describe how software is able to perform a task or make use of information. “It has no understanding” I keep seeing this claim, I don’t understand what it’s supposed to mean. Do we humans have this magical “true understanding”? Can we test for it?

JohnFen 3y ago
I would say that "understanding" something means that you have a mental model of that thing. The more accurate and complete that mental model is, the "deeper" the understanding. LLMs, as I (ahem) understand them, do not have models of the things they're writing about. They do have a model of language (or whatever symbolic system they were trained on), but that's where it ends.
- hammyhavoc 3y ago
  I agree! I asked ChatGPT "Does an LLM understand, or is it simply Bayesian inference?", and the most relevant part of its output: "The model's understanding of language is derived from the vast amount of text it has been trained on. By capturing statistical regularities and patterns, an LLM can generate text that appears to be comprehensible and contextually appropriate. However, it's important to note that LLMs do not possess true understanding or consciousness. They lack semantic or conceptual understanding in the way humans do. Overall, an LLM employs a combination of statistical techniques, including Bayesian inference, and pattern recognition to generate text that simulates understanding to a certain extent."
- circuit10 3y ago
  If it had no model of what it was writing about it wouldn’t be able to write anything coherent. It obviously has a world model, it’s not as advanced as what a human has but it’s there
  - hammyhavoc 3y ago
    The clue is in the name, it's a large language model. When I asked ChatGPT "Does ChatGPT have a world model?", I got this output: "ChatGPT, like other GPT models, does not possess an explicit world model. GPT (Generative Pre-trained Transformer) models, such as ChatGPT, are trained on a large corpus of text data from the internet and do not have direct access to real-time information about the world. They rely solely on the patterns and information they have learned from their training data. While GPT models can generate contextually relevant and coherent responses based on their training, they do not possess up-to-date knowledge or awareness of current events. Their responses are primarily based on patterns, associations, and statistical regularities present in the text they were trained on, rather than an explicit representation of the world. It's important to note that GPT models can occasionally generate plausible-sounding but inaccurate or misleading information, especially when prompted with questions or topics beyond the scope of their training data. Therefore, it's always advisable to verify information from reliable sources when seeking accurate and up-to-date information about the world." When I asked it "Does a large language model infer an understanding of a topic?", it gave this output: "A large language model, such as ChatGPT, can exhibit the appearance of understanding a topic due to its ability to generate coherent and contextually relevant responses. It can make inferences based on patterns and associations learned from its training data, which allows it to generate text that appears knowledgeable about a particular subject. However, it's crucial to understand that this understanding is derived solely from statistical patterns present in the training data, and the model does not possess true comprehension or conceptual understanding like humans do. The model does not have access to real-world experiences, common sense, or the ability to reason in the same way as humans. The responses generated by a language model should be interpreted as a combination of learned patterns and statistical associations rather than as genuine understanding. The model lacks the ability to deeply grasp the meaning, implications, or nuances of a topic beyond what it has learned from the text it was trained on. Therefore, it's essential to approach the responses generated by a language model critically and corroborate information from reliable sources when necessary."
    - circuit10 3y ago
      What does this prove? It proves either that someone trained ChatGPT to spit out this information or that this view is very common on the Internet, which doesn’t necessarily make it right

JohnFen 3y ago

I would say that "understanding" something means that you have a mental model of that thing. The more accurate and complete that mental model is, the "deeper" the understanding. LLMs, as I (ahem) understand them, do not have models of the things they're writing about. They do have a model of language (or whatever symbolic system they were trained on), but that's where it ends.

hammyhavoc 3y ago
I agree! I asked ChatGPT "Does an LLM understand, or is it simply Bayesian inference?", and the most relevant part of its output: "The model's understanding of language is derived from the vast amount of text it has been trained on. By capturing statistical regularities and patterns, an LLM can generate text that appears to be comprehensible and contextually appropriate. However, it's important to note that LLMs do not possess true understanding or consciousness. They lack semantic or conceptual understanding in the way humans do. Overall, an LLM employs a combination of statistical techniques, including Bayesian inference, and pattern recognition to generate text that simulates understanding to a certain extent."
circuit10 3y ago
If it had no model of what it was writing about it wouldn’t be able to write anything coherent. It obviously has a world model, it’s not as advanced as what a human has but it’s there
- hammyhavoc 3y ago
  The clue is in the name, it's a large language model. When I asked ChatGPT "Does ChatGPT have a world model?", I got this output: "ChatGPT, like other GPT models, does not possess an explicit world model. GPT (Generative Pre-trained Transformer) models, such as ChatGPT, are trained on a large corpus of text data from the internet and do not have direct access to real-time information about the world. They rely solely on the patterns and information they have learned from their training data. While GPT models can generate contextually relevant and coherent responses based on their training, they do not possess up-to-date knowledge or awareness of current events. Their responses are primarily based on patterns, associations, and statistical regularities present in the text they were trained on, rather than an explicit representation of the world. It's important to note that GPT models can occasionally generate plausible-sounding but inaccurate or misleading information, especially when prompted with questions or topics beyond the scope of their training data. Therefore, it's always advisable to verify information from reliable sources when seeking accurate and up-to-date information about the world." When I asked it "Does a large language model infer an understanding of a topic?", it gave this output: "A large language model, such as ChatGPT, can exhibit the appearance of understanding a topic due to its ability to generate coherent and contextually relevant responses. It can make inferences based on patterns and associations learned from its training data, which allows it to generate text that appears knowledgeable about a particular subject. However, it's crucial to understand that this understanding is derived solely from statistical patterns present in the training data, and the model does not possess true comprehension or conceptual understanding like humans do. The model does not have access to real-world experiences, common sense, or the ability to reason in the same way as humans. The responses generated by a language model should be interpreted as a combination of learned patterns and statistical associations rather than as genuine understanding. The model lacks the ability to deeply grasp the meaning, implications, or nuances of a topic beyond what it has learned from the text it was trained on. Therefore, it's essential to approach the responses generated by a language model critically and corroborate information from reliable sources when necessary."
  - circuit10 3y ago
    What does this prove? It proves either that someone trained ChatGPT to spit out this information or that this view is very common on the Internet, which doesn’t necessarily make it right