> could the GPT-4 secret sauce be RLHF weighing intelligent answers higher? That part is one of the rare things that the technical report addresses. In Appendi

> could the GPT-4 secret sauce be RLHF weighing intelligent answers higher? That part is one of the rare things that the technical report addresses. In Appendix B[0], they show that RLHF does not improve capabilities on human tasks. It does improve alignment. To me, this is an indication that they performed better scaling analysis and pretrained until it no longer improved. As the Chinchilla paper showed, GPT-3 was undertrained, so any fine-tuning also improved its capabilities. To address your question though, consider two things: first, there are many more ways to be incorrect than to be correct, so even just prediction will find correct answers more likely than incorrect ones. Second, the corpus goes through a significant filtering process; they didn't just feed the raw Twitter firehose to it. [0]: https://arxiv.org/pdf/2303.08774.pdf https://arxiv.org/pdf/2303.08774.pdf (As a side-note, it feels weird to me that they used a free academic archive to store their technical report, even though it cannot go through peer review or be accepted in any academic publication.)