3 ms·

I wonder if you can simply ask ChatGPT to provide a confidence score with each answer it gives.

by dom96 4y ago

rednerrus 4y ago
ChatGPT (gpt3.5-turbo) is terrible at calculating anything. I've seen some preliminary evidence that GPT 4.0 is better at calculation so it may be possible for it.
OscarCunningham 4y ago
My understanding was that ChatGPT simply puts a probability distribution over the next word, so I don't see why it's not as simple as just reporting how high those probabilities were for the answer it gave, relative to whatever would be typical.
- recursivecaveat 4y ago
  Those values are probably not intelligible as confidence scores. For example if it answers a question with "They died in 1902", since there are a lot of euphamisms and rephrasings of 'died' it will get a relatively low probability. 1902 probably gets a high score, but you can't really rely on that since it might just as well be hallucinating and pulled the year from some famous event in that person's life.
yeldarb 4y ago
The GPT-4 paper & post[1] describe that the original model is pretty good at predicting the probability of its own correctness (well-calibrated confidence) but the post-processing degrades this property: > GPT-4 can also be confidently wrong in its predictions, not taking care to double-check work when it’s likely to make a mistake. Interestingly, the base pre-trained model is highly calibrated (its predicted confidence in an answer generally matches the probability of being correct). However, through our current post-training process, the calibration is reduced. > Left: Calibration plot of the pre-trained GPT-4 model on an MMLU subset. The model’s confidence in its prediction closely matches the probability of being correct. The dotted diagonal line represents perfect calibration. Right: Calibration plot of post-trained PPO GPT-4 model on the same MMLU subset. Our current process hurts the calibration quite a bit. [1] https://openai.com/research/gpt-4#:~:text=GPT%2D4%20can%20also%20be%20confidently%20wrong%20in%20its%20predictions https://openai.com/research/gpt-4#:~:text=GPT%2D4%20can%20al...

I wonder if you can simply ask ChatGPT to provide a confidence score with each answer it gives.

by dom96 4y ago

I wonder if you can simply ask ChatGPT to provide a confidence score with each answer it gives.

ChatGPT (gpt3.5-turbo) is terrible at calculating anything. I've seen some preliminary evidence that GPT 4.0 is better at calculation so it may be possible for it.

OscarCunningham 4y ago

My understanding was that ChatGPT simply puts a probability distribution over the next word, so I don't see why it's not as simple as just reporting how high those probabilities were for the answer it gave, relative to whatever would be typical.

recursivecaveat 4y ago
Those values are probably not intelligible as confidence scores. For example if it answers a question with "They died in 1902", since there are a lot of euphamisms and rephrasings of 'died' it will get a relatively low probability. 1902 probably gets a high score, but you can't really rely on that since it might just as well be hallucinating and pulled the year from some famous event in that person's life.

yeldarb 4y ago

The GPT-4 paper & post[1] describe that the original model is pretty good at predicting the probability of its own correctness (well-calibrated confidence) but the post-processing degrades this property: > GPT-4 can also be confidently wrong in its predictions, not taking care to double-check work when it’s likely to make a mistake. Interestingly, the base pre-trained model is highly calibrated (its predicted confidence in an answer generally matches the probability of being correct). However, through our current post-training process, the calibration is reduced. > Left: Calibration plot of the pre-trained GPT-4 model on an MMLU subset. The model’s confidence in its prediction closely matches the probability of being correct. The dotted diagonal line represents perfect calibration. Right: Calibration plot of post-trained PPO GPT-4 model on the same MMLU subset. Our current process hurts the calibration quite a bit. [1] https://openai.com/research/gpt-4#:~:text=GPT%2D4%20can%20also%20be%20confidently%20wrong%20in%20its%20predictions https://openai.com/research/gpt-4#:~:text=GPT%2D4%20can%20al...