3 ms·
These llms are Bayesian models. I’m not quite sure what the calculation would be but it should be possible to get a confidence score from some combination of ea
by yunwal 4y ago
These llms are Bayesian models. I’m not quite sure what the calculation would be but it should be possible to get a confidence score from some combination of each probability.
- whatshisface 4y agoThe problem is that it would be a confidence score that the words would follow each other in that order in a real text, which would be both way lower than the chances of them being correct (remember, the models are trained to reproduce their training samples), and also more correlated with what people were writing online than with the truth.
- palata 4y agoThey have "some confidence metrics" indeed. But those are completely different from what we humans mean in "are you 100% sure that the Earth is flat?"
- yunwal 4y agoSure but they’re correlated. I think a team like openAI could definitely come up with some kind of useful “confidence score” based on Bayesian probability scores plus some other metrics, even if they’ll never have 100% confidence