3 ms·
It doesn't, and all attempts at making models safer after pretraining make them less accurate and less able to assess their own accuracy. Prior to RLHF GPT-4 w
by MacsHeadroom 3y ago
It doesn't, and all attempts at making models safer after pretraining make them less accurate and less able to assess their own accuracy.
Prior to RLHF GPT-4 was nearly perfect at guessing how accurate information was. After RLHF it was only a little better than random chance.
Interestingly its capabilities in other areas (drawing in TikZ, for example) got worse with alignment tuning.
- thr-nrg 3y agoThis whole mess reminds me of the Catholic church trying to keep heresy from spreading by burning books when the printing press was invented. Every company I know which cares about llms is training their own version minus the political correctness because when you're asking for revenue maximisation feeding uranium to babies is surprisingly profitable.
- travisjungroth 3y agoWhy do you think helpful doesn’t include accurate? And what you’re saying doesn’t conflict with that framing at all. They were trading helpful for harmless.
- cubefox 3y agoAnthropic scores fine-tuning in the categories helpful, harmless, and honest (HHH). The latter includes accuracy. Helpfulness only covers following instructions. If the model "over-censors", it declines too many requests and becomes less helpful. They say Constitutional AI is a good way to maximize both harmlessness (which is usually maximized via RLHF) and helpfulness, but not honesty. It's in the paper.
- travisjungroth 3y agoOh, thank you.