5 ms·

Something which should happen: AI productions must be allowed to express doubt, levels of low confidence and the like. In a (hypothetical) 1960s SciFi film a k

Something which should happen: AI productions must be allowed to express doubt, levels of low confidence and the like. In a (hypothetical) 1960s SciFi film a knowledge system may have responded like this: > assumption: subject deceased > confidence score: 0.0000020017 > sources: none In contrast to this, current systems are tailored to respond with high confidence: > I'm sure X died in 2018. > It has been written about in the New York Times, The Guardian, Le Monde. > Here is a (hallucinated) link: https://www.theguardian.com/obituaries/x-obituary (Compare various related stories.)

dom96 4y ago
I wonder if you can simply ask ChatGPT to provide a confidence score with each answer it gives.
- rednerrus 4y ago
  ChatGPT (gpt3.5-turbo) is terrible at calculating anything. I've seen some preliminary evidence that GPT 4.0 is better at calculation so it may be possible for it.
- OscarCunningham 4y ago
  My understanding was that ChatGPT simply puts a probability distribution over the next word, so I don't see why it's not as simple as just reporting how high those probabilities were for the answer it gave, relative to whatever would be typical.
  - recursivecaveat 4y ago
    Those values are probably not intelligible as confidence scores. For example if it answers a question with "They died in 1902", since there are a lot of euphamisms and rephrasings of 'died' it will get a relatively low probability. 1902 probably gets a high score, but you can't really rely on that since it might just as well be hallucinating and pulled the year from some famous event in that person's life.
- yeldarb 4y ago
  The GPT-4 paper & post[1] describe that the original model is pretty good at predicting the probability of its own correctness (well-calibrated confidence) but the post-processing degrades this property: > GPT-4 can also be confidently wrong in its predictions, not taking care to double-check work when it’s likely to make a mistake. Interestingly, the base pre-trained model is highly calibrated (its predicted confidence in an answer generally matches the probability of being correct). However, through our current post-training process, the calibration is reduced. > Left: Calibration plot of the pre-trained GPT-4 model on an MMLU subset. The model’s confidence in its prediction closely matches the probability of being correct. The dotted diagonal line represents perfect calibration. Right: Calibration plot of post-trained PPO GPT-4 model on the same MMLU subset. Our current process hurts the calibration quite a bit. [1] https://openai.com/research/gpt-4#:~:text=GPT%2D4%20can%20also%20be%20confidently%20wrong%20in%20its%20predictions https://openai.com/research/gpt-4#:~:text=GPT%2D4%20can%20al...
EarthLaunch 4y ago
I love this because I want more emphasis on this between humans. I hear there are languages with built in confidence expression.
- orange_fritter 4y ago
  Yep, I'm not a language expert but I remember a South American language, maybe related to Quechua, where a sentences contain a syllable that indicates "I am repeating hearsay, I don't know if this is true". Pretty cool.
  - masswerk 4y ago
    This would be an unexpected outcome: AI productions motivating us to generally move towards prefix/postfix/infix decorations. "X studied at uhm-Oxford." -> I'm not sure, if it was really Oxford, but I have picked this up somewhere. But I'm sure, X did go to university. (As opposed to, "X uhm-studied at uhm-Oxford": I think, X has some higher education and it may have well been at Oxford.) :-)

dom96 4y ago

I wonder if you can simply ask ChatGPT to provide a confidence score with each answer it gives.

rednerrus 4y ago
ChatGPT (gpt3.5-turbo) is terrible at calculating anything. I've seen some preliminary evidence that GPT 4.0 is better at calculation so it may be possible for it.
OscarCunningham 4y ago
My understanding was that ChatGPT simply puts a probability distribution over the next word, so I don't see why it's not as simple as just reporting how high those probabilities were for the answer it gave, relative to whatever would be typical.
- recursivecaveat 4y ago
  Those values are probably not intelligible as confidence scores. For example if it answers a question with "They died in 1902", since there are a lot of euphamisms and rephrasings of 'died' it will get a relatively low probability. 1902 probably gets a high score, but you can't really rely on that since it might just as well be hallucinating and pulled the year from some famous event in that person's life.
yeldarb 4y ago
The GPT-4 paper & post[1] describe that the original model is pretty good at predicting the probability of its own correctness (well-calibrated confidence) but the post-processing degrades this property: > GPT-4 can also be confidently wrong in its predictions, not taking care to double-check work when it’s likely to make a mistake. Interestingly, the base pre-trained model is highly calibrated (its predicted confidence in an answer generally matches the probability of being correct). However, through our current post-training process, the calibration is reduced. > Left: Calibration plot of the pre-trained GPT-4 model on an MMLU subset. The model’s confidence in its prediction closely matches the probability of being correct. The dotted diagonal line represents perfect calibration. Right: Calibration plot of post-trained PPO GPT-4 model on the same MMLU subset. Our current process hurts the calibration quite a bit. [1] https://openai.com/research/gpt-4#:~:text=GPT%2D4%20can%20also%20be%20confidently%20wrong%20in%20its%20predictions https://openai.com/research/gpt-4#:~:text=GPT%2D4%20can%20al...

EarthLaunch 4y ago

I love this because I want more emphasis on this between humans. I hear there are languages with built in confidence expression.

orange_fritter 4y ago
Yep, I'm not a language expert but I remember a South American language, maybe related to Quechua, where a sentences contain a syllable that indicates "I am repeating hearsay, I don't know if this is true". Pretty cool.
- masswerk 4y ago
  This would be an unexpected outcome: AI productions motivating us to generally move towards prefix/postfix/infix decorations. "X studied at uhm-Oxford." -> I'm not sure, if it was really Oxford, but I have picked this up somewhere. But I'm sure, X did go to university. (As opposed to, "X uhm-studied at uhm-Oxford": I think, X has some higher education and it may have well been at Oxford.) :-)