4 ms·
Auto Label can do it even with closed-source LLMs: https://www.refuel.ai/blog-posts/labeling-with-confidence https://www.refuel.ai/blog-posts/labeling-with-conf
by simbolit 3y ago
Auto Label can do it even with closed-source LLMs: https://www.refuel.ai/blog-posts/labeling-with-confidence https://www.refuel.ai/blog-posts/labeling-with-confidence
Also, your 90% number is a 'plausible-looking hallucination', which is mildly funny.
- unraveller 3y ago>Explicitly prompting the LLM to output a confidence score is the least accurate They don't show the whole prompt they use for this so I am sceptical of the comparison that recommends their service which does the token level accuracy. In my short testing you can drill down some acceptable accuracy for specific domains if you lead it over some general "ask yourself" questions then "finally, output a % confidence"
- simbolit 3y agoI couldn't care less about the specific implementation. My parent comment said "LLMs are useless for classification, because they can't give confidence scores". I think i have shown that to be demonstrably false.
- fnordpiglet 3y agoMy understanding is these are more “confidence in the tokens selected being appropriate” rather than the more fundamental “confidence the classification chosen is accurate given training data,” where training data is samples of the distribution being classified rather than language corpus. They seem similar at first blush but the first is more about appropriateness of phrasing and how esoteric the semantics are, and the latter more descriptive of some statistically observable process - which is something LLMs are literally incapable of reasoning about directly.