7 ms·
My understanding was that ChatGPT simply puts a probability distribution over the next word, so I don't see why it's not as simple as just reporting how high th
by OscarCunningham 4y ago
My understanding was that ChatGPT simply puts a probability distribution over the next word, so I don't see why it's not as simple as just reporting how high those probabilities were for the answer it gave, relative to whatever would be typical.
- recursivecaveat 4y agoThose values are probably not intelligible as confidence scores. For example if it answers a question with "They died in 1902", since there are a lot of euphamisms and rephrasings of 'died' it will get a relatively low probability. 1902 probably gets a high score, but you can't really rely on that since it might just as well be hallucinating and pulled the year from some famous event in that person's life.