4 ms·
I asked it "What is the accuracy and confidence level (you must rate it out of 10) that the above answer is correct ?" It always generates a random number when
by tap-snap-or-nap 4y ago
I asked it "What is the accuracy and confidence level (you must rate it out of 10) that the above answer is correct ?"
It always generates a random number when it is caught making up bullshit sometimes claims that the information 10/10 accurate and LLM's always provides accurate information.
- sorz 4y agoOpenAI did exact the test for GPT-4. The raw, non-fine-tune GPT-4 is quite good at predicting confidence level ("highly calibrated" by their words). But the RLHF fine-tuning process seems ruin its calibration. Figure 8 on page 12 of GPT-4 Technical Report shows this dramatic changes before & after fine-tuning.