3 ms·

Except we can’t get LLMs to say it doesn’t know - it doesn’t process the concept of “know” and “don’t know”, heck it doesn’t even process the concept of true an

Except we can’t get LLMs to say it doesn’t know - it doesn’t process the concept of “know” and “don’t know”, heck it doesn’t even process the concept of true and false. It’s a language model. It models the statistical properties of human languages.

vidarh 3y ago
> Except we can’t get LLMs to say it doesn’t know Here is a direct pair of quotes from a recent conversation with GPT4: > Me: Answer in a single sentence, please. Do you, or do you not know the contents of Dr. Franz 1994 paper "Code Generation on the Fly: A key to portable software"? > ChatGPT: No, I do not know the contents of Dr. Franz's 1994 paper "Code Generation on the Fly: A Key to Portable Software." Is it prone to being evasive and waffling and not wanting to admit when it does not know, and preferring to jump to conclusions and try to get away with generalities, yes. Asking about this paper is one of my repeated test cases because it does so badly (there are few online sources on it, but his paper is online; other than that one of my blog posts and the Wikipedia article make up the bulk of text about it). And so it did when I tried it last - it waffled on about the (unrelated) concept of semantic encoding from NLP. As far as I can tell, it is correct: It does not know the contents of the paper. It barely understands the high-level concepts involved ( > it doesn’t process the concept of “know” and “don’t know”, heck it doesn’t even process the concept of true and false. > It’s a language model. It models the statistical properties of human languages. It can explain them and use them. We don't know enough about what "knowing" something or "processing" a concept means in terms of human thought processes to know whether there's a meaningful distinction between the level at which an LLM processes these concepts vs. humans or whether there is a meaningful distinction between reasoning and intelligence vs. "modelling the statistical properties of human languages".
- TerrifiedMouse 3y ago
  > As far as I can tell, it is correct: It does not know the contents of the paper. It barely understands the high-level concepts involved LLMs work by predicting the next bunch of words - it's advanced auto complete. If most of the training data replies "I don't" to the question 'Do you, or do you not know the contents of Dr. Franz 1994 paper "Code Generation on the Fly: A key to portable software"?' then that's what the LLM will say. It's half useful for answering frequently asked questions but don't expect it to evaluate its current state and give you an accurate answer. > We don't know enough about what "knowing" something or "processing" a concept means in terms of human thought processes to know whether there's a meaningful distinction between the level at which an LLM processes these concepts vs. humans or whether there is a meaningful distinction between reasoning and intelligence vs. "modelling the statistical properties of human languages". But we do know what LLMs do, model language. Not knowledge. Not “thought”. Language. And the way we get them to spit out output that’s satisfactory to us is just absurd. If you read the link I posted earlier, you will know that if you set the “temperature” of an LLM to zero, it just repeats itself talking in circles. It’s only by adding randomness to its “next token” search that we get output that possibly satisfactory.

vidarh 3y ago

> Except we can’t get LLMs to say it doesn’t know Here is a direct pair of quotes from a recent conversation with GPT4: > Me: Answer in a single sentence, please. Do you, or do you not know the contents of Dr. Franz 1994 paper "Code Generation on the Fly: A key to portable software"? > ChatGPT: No, I do not know the contents of Dr. Franz's 1994 paper "Code Generation on the Fly: A Key to Portable Software." Is it prone to being evasive and waffling and not wanting to admit when it does not know, and preferring to jump to conclusions and try to get away with generalities, yes. Asking about this paper is one of my repeated test cases because it does so badly (there are few online sources on it, but his paper is online; other than that one of my blog posts and the Wikipedia article make up the bulk of text about it). And so it did when I tried it last - it waffled on about the (unrelated) concept of semantic encoding from NLP. As far as I can tell, it is correct: It does not know the contents of the paper. It barely understands the high-level concepts involved ( > it doesn’t process the concept of “know” and “don’t know”, heck it doesn’t even process the concept of true and false. > It’s a language model. It models the statistical properties of human languages. It can explain them and use them. We don't know enough about what "knowing" something or "processing" a concept means in terms of human thought processes to know whether there's a meaningful distinction between the level at which an LLM processes these concepts vs. humans or whether there is a meaningful distinction between reasoning and intelligence vs. "modelling the statistical properties of human languages".

TerrifiedMouse 3y ago
> As far as I can tell, it is correct: It does not know the contents of the paper. It barely understands the high-level concepts involved LLMs work by predicting the next bunch of words - it's advanced auto complete. If most of the training data replies "I don't" to the question 'Do you, or do you not know the contents of Dr. Franz 1994 paper "Code Generation on the Fly: A key to portable software"?' then that's what the LLM will say. It's half useful for answering frequently asked questions but don't expect it to evaluate its current state and give you an accurate answer. > We don't know enough about what "knowing" something or "processing" a concept means in terms of human thought processes to know whether there's a meaningful distinction between the level at which an LLM processes these concepts vs. humans or whether there is a meaningful distinction between reasoning and intelligence vs. "modelling the statistical properties of human languages". But we do know what LLMs do, model language. Not knowledge. Not “thought”. Language. And the way we get them to spit out output that’s satisfactory to us is just absurd. If you read the link I posted earlier, you will know that if you set the “temperature” of an LLM to zero, it just repeats itself talking in circles. It’s only by adding randomness to its “next token” search that we get output that possibly satisfactory.