3 ms·

How can a neural network evaluate "confidence"? The parameters don't store any information about what inputs were seen in the training data (vs being interpola

How can a neural network evaluate "confidence"? The parameters don't store any information about what inputs were seen in the training data (vs being interpolated) or how accurate the predictions were for those specific inputs. And even if they did, the training data was usually gathered voraciously, without much preference for quality reasoning.

habitue 3y ago
I don't know for sure, but here's a plausible mechanism for how: Multiple sub-networks detect the same pattern in different ways, and confidence is the percent of those sub-networks that fire for a particular instance. There's a ton of overlap and redundancy with so many weights, so there are lots of ways this could work
- brookst 3y ago
  That’s good. Also maybe an architecture that runs the query through multiple times and then evaluates similarity of responses, then selects (or creates) the most-generated one, along with a confidence level of how many of the individual responses were aligned.
montjoy 3y ago
IIRC confidence in video is related to predicting what happens next vs what actually happens. If the two seem to correlate to the model it would give it a higher confidence ranking, which would then be used further for self-reinforced learning.
pbhjpbhj 3y ago
Phind gives me ChatGPT answers with relatively authoritative references to works on the web that (usually!) support the answer. Could it have a post-filter to fact check against the references? I guess that is a slight variation of the sibling (@habitue's) answer; both are checks against external material. I wonder if best resources could be catalogued as the corpus is processed, giving a document vector space to select resources for such 'sense' checking.
XenophileJKO 3y ago
Actually you can get a very good proxy by looking at the probability distrobution of the "answer" tokens. The key here is you have to be able to identify the "answer" tokens. https://arxiv.org/abs/2402.10200 https://arxiv.org/abs/2402.10200

habitue 3y ago

I don't know for sure, but here's a plausible mechanism for how: Multiple sub-networks detect the same pattern in different ways, and confidence is the percent of those sub-networks that fire for a particular instance. There's a ton of overlap and redundancy with so many weights, so there are lots of ways this could work

brookst 3y ago
That’s good. Also maybe an architecture that runs the query through multiple times and then evaluates similarity of responses, then selects (or creates) the most-generated one, along with a confidence level of how many of the individual responses were aligned.

montjoy 3y ago

IIRC confidence in video is related to predicting what happens next vs what actually happens. If the two seem to correlate to the model it would give it a higher confidence ranking, which would then be used further for self-reinforced learning.

pbhjpbhj 3y ago

Phind gives me ChatGPT answers with relatively authoritative references to works on the web that (usually!) support the answer. Could it have a post-filter to fact check against the references? I guess that is a slight variation of the sibling (@habitue's) answer; both are checks against external material. I wonder if best resources could be catalogued as the corpus is processed, giving a document vector space to select resources for such 'sense' checking.

XenophileJKO 3y ago

Actually you can get a very good proxy by looking at the probability distrobution of the "answer" tokens. The key here is you have to be able to identify the "answer" tokens. https://arxiv.org/abs/2402.10200 https://arxiv.org/abs/2402.10200