3 ms·
It doesn't. Accuracy is a comparison between text and the real world. LLMs have no access to the real world. Also RLHF screws up the logprobs.
by smeagull 3y ago
It doesn't. Accuracy is a comparison between text and the real world. LLMs have no access to the real world.
Also RLHF screws up the logprobs.
- deleted 3y ago[deleted]
- mike_hearn 3y agoIt's possible to make accurate predictions about things you can't directly measure, and also know you're accurate. I don't think the word necessarily implies that you need access to the real world. Example: Stephen Hawking had very little access to the real world but could still make accurate predictions with high confidence. How do you mean it screws up the logprobs?
- espadrine 3y agoI believe they reference Figure 8 from the GPT-4 technical report[0], which shows that the pretrained model’s output probability for each answer (a, b, c, or d) is proportional to the probability of being correct, while after PPO (RLHF), it is quite a bit flatter. [0]: https://cdn.openai.com/papers/gpt-4.pdf https://cdn.openai.com/papers/gpt-4.pdf
- mike_hearn 3y agoI see, thanks. It's remarkable that the RLHF has such a drastically negative impact on the model's understanding of the world. Guess that explains the degrading unicorn problem. It makes me wonder how much better at coding an instruct-trained but non-aligned AI would get.