3 ms·

> What I actually suggested is that a willingness to fill in details is necessary, not that filling it in no matter what or without having something reasonable

> What I actually suggested is that a willingness to fill in details is necessary, not that filling it in no matter what or without having something reasonable to fill in is necessary. Except that's exactly what LLMs do due to the way they work. It's statistically guessing the next token. Unfortunately, just because statistically the next token is among the most likely to appear doesn't mean the sentence it forms is correct or even makes any sense. Frankly, the whole way LLMs work is kind of absurd in the light of what we are trying to do. https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/ https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-... The darn algorithm actually has randomness built into it - and it's an absolutely necessary component.

vidarh 3y ago
It needs to fill in something but that something can be an "out of context" admission that it does not know, it does not need to be low probability nonsense about the subject, and that was my point. Sufficient reinforcement that "I don't know" should be the most likely answer in some contexts is certainly important for intelligence.
- TerrifiedMouse 3y ago
  Except we can’t get LLMs to say it doesn’t know - it doesn’t process the concept of “know” and “don’t know”, heck it doesn’t even process the concept of true and false. It’s a language model. It models the statistical properties of human languages.
  - vidarh 3y ago
    > Except we can’t get LLMs to say it doesn’t know Here is a direct pair of quotes from a recent conversation with GPT4: > Me: Answer in a single sentence, please. Do you, or do you not know the contents of Dr. Franz 1994 paper "Code Generation on the Fly: A key to portable software"? > ChatGPT: No, I do not know the contents of Dr. Franz's 1994 paper "Code Generation on the Fly: A Key to Portable Software." Is it prone to being evasive and waffling and not wanting to admit when it does not know, and preferring to jump to conclusions and try to get away with generalities, yes. Asking about this paper is one of my repeated test cases because it does so badly (there are few online sources on it, but his paper is online; other than that one of my blog posts and the Wikipedia article make up the bulk of text about it). And so it did when I tried it last - it waffled on about the (unrelated) concept of semantic encoding from NLP. As far as I can tell, it is correct: It does not know the contents of the paper. It barely understands the high-level concepts involved ( > it doesn’t process the concept of “know” and “don’t know”, heck it doesn’t even process the concept of true and false. > It’s a language model. It models the statistical properties of human languages. It can explain them and use them. We don't know enough about what "knowing" something or "processing" a concept means in terms of human thought processes to know whether there's a meaningful distinction between the level at which an LLM processes these concepts vs. humans or whether there is a meaningful distinction between reasoning and intelligence vs. "modelling the statistical properties of human languages".
    - TerrifiedMouse 3y ago
      > As far as I can tell, it is correct: It does not know the contents of the paper. It barely understands the high-level concepts involved LLMs work by predicting the next bunch of words - it's advanced auto complete. If most of the training data replies "I don't" to the question 'Do you, or do you not know the contents of Dr. Franz 1994 paper "Code Generation on the Fly: A key to portable software"?' then that's what the LLM will say. It's half useful for answering frequently asked questions but don't expect it to evaluate its current state and give you an accurate answer. > We don't know enough about what "knowing" something or "processing" a concept means in terms of human thought processes to know whether there's a meaningful distinction between the level at which an LLM processes these concepts vs. humans or whether there is a meaningful distinction between reasoning and intelligence vs. "modelling the statistical properties of human languages". But we do know what LLMs do, model language. Not knowledge. Not “thought”. Language. And the way we get them to spit out output that’s satisfactory to us is just absurd. If you read the link I posted earlier, you will know that if you set the “temperature” of an LLM to zero, it just repeats itself talking in circles. It’s only by adding randomness to its “next token” search that we get output that possibly satisfactory.

vidarh 3y ago

It needs to fill in something but that something can be an "out of context" admission that it does not know, it does not need to be low probability nonsense about the subject, and that was my point. Sufficient reinforcement that "I don't know" should be the most likely answer in some contexts is certainly important for intelligence.

TerrifiedMouse 3y ago
Except we can’t get LLMs to say it doesn’t know - it doesn’t process the concept of “know” and “don’t know”, heck it doesn’t even process the concept of true and false. It’s a language model. It models the statistical properties of human languages.
- vidarh 3y ago
  > Except we can’t get LLMs to say it doesn’t know Here is a direct pair of quotes from a recent conversation with GPT4: > Me: Answer in a single sentence, please. Do you, or do you not know the contents of Dr. Franz 1994 paper "Code Generation on the Fly: A key to portable software"? > ChatGPT: No, I do not know the contents of Dr. Franz's 1994 paper "Code Generation on the Fly: A Key to Portable Software." Is it prone to being evasive and waffling and not wanting to admit when it does not know, and preferring to jump to conclusions and try to get away with generalities, yes. Asking about this paper is one of my repeated test cases because it does so badly (there are few online sources on it, but his paper is online; other than that one of my blog posts and the Wikipedia article make up the bulk of text about it). And so it did when I tried it last - it waffled on about the (unrelated) concept of semantic encoding from NLP. As far as I can tell, it is correct: It does not know the contents of the paper. It barely understands the high-level concepts involved ( > it doesn’t process the concept of “know” and “don’t know”, heck it doesn’t even process the concept of true and false. > It’s a language model. It models the statistical properties of human languages. It can explain them and use them. We don't know enough about what "knowing" something or "processing" a concept means in terms of human thought processes to know whether there's a meaningful distinction between the level at which an LLM processes these concepts vs. humans or whether there is a meaningful distinction between reasoning and intelligence vs. "modelling the statistical properties of human languages".
  - TerrifiedMouse 3y ago
    > As far as I can tell, it is correct: It does not know the contents of the paper. It barely understands the high-level concepts involved LLMs work by predicting the next bunch of words - it's advanced auto complete. If most of the training data replies "I don't" to the question 'Do you, or do you not know the contents of Dr. Franz 1994 paper "Code Generation on the Fly: A key to portable software"?' then that's what the LLM will say. It's half useful for answering frequently asked questions but don't expect it to evaluate its current state and give you an accurate answer. > We don't know enough about what "knowing" something or "processing" a concept means in terms of human thought processes to know whether there's a meaningful distinction between the level at which an LLM processes these concepts vs. humans or whether there is a meaningful distinction between reasoning and intelligence vs. "modelling the statistical properties of human languages". But we do know what LLMs do, model language. Not knowledge. Not “thought”. Language. And the way we get them to spit out output that’s satisfactory to us is just absurd. If you read the link I posted earlier, you will know that if you set the “temperature” of an LLM to zero, it just repeats itself talking in circles. It’s only by adding randomness to its “next token” search that we get output that possibly satisfactory.