4 ms·

I would really like for LLMs to reach a point where they can actually tell you if they're not quite sure about something, rather than confidently outputting the

I would really like for LLMs to reach a point where they can actually tell you if they're not quite sure about something, rather than confidently outputting their BS like it's fact. Too many people interpret LLM output as though it's guaranteed correct. Certainly there's some confidence value metric that they can understand internally and use when wording their response.

munchler 3y ago
The irony of this comment is rich. There’s no such “confidence value metric” known, no matter how “certain” you are. This is why techniques like chain-of-thought or self-correction are used instead. LLM’s predict the next token in a sequence. That’s it. The final softmax layer gives a per-token probability, but that is not the same thing as overall confidence in the response.
- famouswaffles 3y ago
  The irony of this comment is rich. >The final softmax layer gives a per-token probability, but that is not the same thing as overall confidence in the response. Clearly not. https://imgur.com/a/3gYel9r https://imgur.com/a/3gYel9r
  - munchler 3y ago
    Hah, I wasn’t aware of that. Does it say how the base model computes its confidence? Is it requested in the prompt?
    - famouswaffles 3y ago
      They didn't specify.
  - singularity2001 3y ago
    This may be philosophically one of the most important graphs regarding the current state of AI! It shows that large language models already DO have an internal model of confidence in their statements, all that is needed is to train the models to output this confidence alongside their hallucinations if the confidence is low.
- layer8 3y ago
  This might just mean that they have been trained wrong. Because they should (in order to be useful) “predict” a response consistent with the knowledge or lack of knowledge encoded in the network. When a human has read (has been “trained” on) some corpus of sources, they have a feel for their confidence in particular statements about topics treated in those sources, based on he content of those sources. There’s no inherent reason why an LLM shouldn’t likewise be able to internally determine a level of confidence, and produce (“predict”) answers consistent with that.
  - roflyear 3y ago
    This is the issue with saying LLMs have understanding: it leads to misunderstanding how these systems work. Which, can be problematic! They don't work like people, so diving into that isn't productive.
roflyear 3y ago
How can they do this? What would be the mechanism for an LLM to know what is the right data and what isn't the right data?

munchler 3y ago

The irony of this comment is rich. There’s no such “confidence value metric” known, no matter how “certain” you are. This is why techniques like chain-of-thought or self-correction are used instead. LLM’s predict the next token in a sequence. That’s it. The final softmax layer gives a per-token probability, but that is not the same thing as overall confidence in the response.

famouswaffles 3y ago
The irony of this comment is rich. >The final softmax layer gives a per-token probability, but that is not the same thing as overall confidence in the response. Clearly not. https://imgur.com/a/3gYel9r https://imgur.com/a/3gYel9r
- munchler 3y ago
  Hah, I wasn’t aware of that. Does it say how the base model computes its confidence? Is it requested in the prompt?
  - famouswaffles 3y ago
    They didn't specify.
- singularity2001 3y ago
  This may be philosophically one of the most important graphs regarding the current state of AI! It shows that large language models already DO have an internal model of confidence in their statements, all that is needed is to train the models to output this confidence alongside their hallucinations if the confidence is low.
layer8 3y ago
This might just mean that they have been trained wrong. Because they should (in order to be useful) “predict” a response consistent with the knowledge or lack of knowledge encoded in the network. When a human has read (has been “trained” on) some corpus of sources, they have a feel for their confidence in particular statements about topics treated in those sources, based on he content of those sources. There’s no inherent reason why an LLM shouldn’t likewise be able to internally determine a level of confidence, and produce (“predict”) answers consistent with that.
- roflyear 3y ago
  This is the issue with saying LLMs have understanding: it leads to misunderstanding how these systems work. Which, can be problematic! They don't work like people, so diving into that isn't productive.

roflyear 3y ago

How can they do this? What would be the mechanism for an LLM to know what is the right data and what isn't the right data?