I think Figure 8 in the GPT4 report already implies this for RLHF, i.e. the base model is fairly well-calibrated about its confidence but the RLHFed one becomes

I think Figure 8 in the GPT4 report already implies this for RLHF, i.e. the base model is fairly well-calibrated about its confidence but the RLHFed one becomes less calibrated which ought to make confident, incorrect statements worse. Idk why they don't discuss that more in-depth.