3 ms·

Large language models have read everything, and they don't know anything. They are excellent imitators, being able to clone the style and contents of any subje

Large language models have read everything, and they don't know anything. They are excellent imitators, being able to clone the style and contents of any subject or source you ask for. When you prompt them, they will uncritically generate a text that combines the relevant topics in creative ways, without the least understanding of their meaning. Their original training causes them to memorize lots of concepts, both high and low level, so they can apply them while generating new content. But they have no reception or self-assessment of what they are creating.

chaxor 3y ago
Can you prove that it actually "doesn't know anything"? What do you mean by that? Being critical does not make you educated on the subject. There are so many comments like this, yet never provide any useful information. Saying there's no value in something, as everyone seems to try to do regarding LLMs, should come with more novel insights than parroting this same idea along with every single person on HN.
- sltkr 3y ago
  Especially ironic that the original comment came from a user named "TuringTest"!
  - TuringTest 3y ago
    Yup, I've been thinking about this topic since long before large language model existed :P
- j0057 3y ago
  That's easy: ask it anything, then "correct" it with some outrageous nonsense. It will apologize (as if to express regret), and say you're correct, and now the conversation is poisoned with whatever nonsense you fed it. All form and zero substance. We fall for it because normally the use of language is an expression of something, with ChatGPT language is just that, language, with no meaning. To me that proves knowing and reasoning happens on a deeper, more symbolic level and language is an expression of that, as are other things.
  - chaxor 3y ago
    This isn't always true, especially not with gpt-4. Also, this isn't a proof really, or even evidence to show that it doesn't 'know' something. It appears to reason well about many tasks - specifically "under the hood" (reasoning not explicitly stated within the output provided). Yes, of course "it's just a language model" is spouted over and over and is sometimes true (though obviously not for gpt-4), but that statement does not provide any insight at all, and it certainly does not necessarily limit the capability of 'obtaining knowledge'.
    - TuringTest 3y ago
      'Reasoning' (as in deriving new statements with precision, following logical inference rules) is precisely what the large language models can't do. It's better to think of this models as 'generating' chains of relevant words, where 'relevant' is defined by similarity of those areas of knowledge on which it has been trained, and which are "activated" as close to the topics in the prompt. Which is not at all dissimilar to how humans learn about a new topic, btw. This way, by "activating" concepts of areas of knowledge and finding words that are more likely than others to fit those concepts, the model is able to create texts following the constraints you instruct it with - such as poems that rhyme, or critical analysis of scientific articles. The most important point to be aware of is that this creation model is completely different to how automatic reasoning models create content, which is by having a formal representation of a knowledge domain and creating logical inferences that can be mathematically proven correct within the model. A reasoning model cannot lie, but it cannot create content beyond the logical implications of its premises; its quite the opposite of what language models do.
- TuringTest 3y ago
  I have not said there is no value in LLMs, quite the contrary. What I'm warning is against thinking of them as independent agents with their own minds, because they don't work like that at all, so you'd be anthropomorphising them. These models certainly have a compilation of knowledge, but it is statistical knowledge - in the same way as a book of logarithms has lots of mathematical knowledge, but you wouldn't say that the book 'knows logarithms'. The compilation contains statistical 'truths' about the topics on which it has been trained; and contrary to a written book, that knowledge can be used operationally to build new information. Yet that static knowledge does not reach the point of having a will of its own; there is nothing in the content generation system that makes it take decisions or establish its own objectives from its statistical tables of compiled knowledge.

chaxor 3y ago

Can you prove that it actually "doesn't know anything"? What do you mean by that? Being critical does not make you educated on the subject. There are so many comments like this, yet never provide any useful information. Saying there's no value in something, as everyone seems to try to do regarding LLMs, should come with more novel insights than parroting this same idea along with every single person on HN.

sltkr 3y ago
Especially ironic that the original comment came from a user named "TuringTest"!
- TuringTest 3y ago
  Yup, I've been thinking about this topic since long before large language model existed :P
j0057 3y ago
That's easy: ask it anything, then "correct" it with some outrageous nonsense. It will apologize (as if to express regret), and say you're correct, and now the conversation is poisoned with whatever nonsense you fed it. All form and zero substance. We fall for it because normally the use of language is an expression of something, with ChatGPT language is just that, language, with no meaning. To me that proves knowing and reasoning happens on a deeper, more symbolic level and language is an expression of that, as are other things.
- chaxor 3y ago
  This isn't always true, especially not with gpt-4. Also, this isn't a proof really, or even evidence to show that it doesn't 'know' something. It appears to reason well about many tasks - specifically "under the hood" (reasoning not explicitly stated within the output provided). Yes, of course "it's just a language model" is spouted over and over and is sometimes true (though obviously not for gpt-4), but that statement does not provide any insight at all, and it certainly does not necessarily limit the capability of 'obtaining knowledge'.
  - TuringTest 3y ago
    'Reasoning' (as in deriving new statements with precision, following logical inference rules) is precisely what the large language models can't do. It's better to think of this models as 'generating' chains of relevant words, where 'relevant' is defined by similarity of those areas of knowledge on which it has been trained, and which are "activated" as close to the topics in the prompt. Which is not at all dissimilar to how humans learn about a new topic, btw. This way, by "activating" concepts of areas of knowledge and finding words that are more likely than others to fit those concepts, the model is able to create texts following the constraints you instruct it with - such as poems that rhyme, or critical analysis of scientific articles. The most important point to be aware of is that this creation model is completely different to how automatic reasoning models create content, which is by having a formal representation of a knowledge domain and creating logical inferences that can be mathematically proven correct within the model. A reasoning model cannot lie, but it cannot create content beyond the logical implications of its premises; its quite the opposite of what language models do.
TuringTest 3y ago
I have not said there is no value in LLMs, quite the contrary. What I'm warning is against thinking of them as independent agents with their own minds, because they don't work like that at all, so you'd be anthropomorphising them. These models certainly have a compilation of knowledge, but it is statistical knowledge - in the same way as a book of logarithms has lots of mathematical knowledge, but you wouldn't say that the book 'knows logarithms'. The compilation contains statistical 'truths' about the topics on which it has been trained; and contrary to a written book, that knowledge can be used operationally to build new information. Yet that static knowledge does not reach the point of having a will of its own; there is nothing in the content generation system that makes it take decisions or establish its own objectives from its statistical tables of compiled knowledge.