2 ms·
> its confidence scores important to note that the "confidence" score is... maybe not what people think it is - kind of useless, and just a convenience step fr
by preommr 6d ago
> its confidence scores
important to note that the "confidence" score is... maybe not what people think it is - kind of useless, and just a convenience step from the probabilities.
from the docs: "confidence is a statistic computed from the probability distribution the answer already gives you." [0] I actually encourage people to visit the docs because it has a specific page on this with a little applet to really make this clear.
[0] https://docs.typesafe.ai/confidence https://docs.typesafe.ai/confidence
- wongarsu 5d agoWhat else do people think it is? If Typesafe had found a way to measure arbitrary AI results against objective reality (past, future and present) they'd either be making a killing on the stock market or working for the NRO, not publishing that as a confidence value on their API
- preommr 5d agoWell I think there's an expectation that it's similar to the probability score, something that's outputted by the model itself, and so there's some level of "intelligence" (e.g. being able to recognize if the subject matter is relevant to the data it's been trained on, or as an accumulation of errors e.g. it couldn't figure out the question).