4 ms·
> GPT-3 was available for years This is a common misunderstanding. ChatGPT launched with GPT-3.5 (not GPT-3) and was the first model to have RLHF. GPT-3.5 over
by mritchie712 3y ago
> GPT-3 was available for years
This is a common misunderstanding. ChatGPT launched with GPT-3.5 (not GPT-3) and was the first model to have RLHF. GPT-3.5 over the API was noticeably better at most tasks then GPT-3.
- simonw 3y agoThat's not quite accurate: InstructGPT was an earlier thing that made GPT-3 much easier to use (it could answer questions rather than just deal in completion prompts), and that was exposed through the GPT-3 API for quite around 11 months before ChatGPT was released. https://openai.com/research/instruction-following https://openai.com/research/instruction-following is from January 2022 "These InstructGPT models, which are trained with humans in the loop, are now deployed as the default language models on our API."
- mritchie712 3y agoYou're right, I shouldn't have said "first" there. Instruct had RLHF. But I don't think ChatGPT would have worked nearly as well using InstructGPT as the model. GPT-3.5 was still a better model, especially for chat, than InstructGPT. > We trained this model using Reinforcement Learning from Human Feedback (RLHF), using the same methods as InstructGPT, but with slight differences in the data collection setup. We trained an initial model using supervised fine-tuning: human AI trainers provided conversations in which they played both sides—the user and an AI assistant. We gave the trainers access to model-written suggestions to help them compose their responses. We mixed this new dialogue dataset with the InstructGPT dataset, which we transformed into a dialogue format. https://openai.com/blog/chatgpt https://openai.com/blog/chatgpt
- simonw 3y agoYeah, you make a great argument. I agree, ChatGPT without the extra RLHF would likely have had much less of an impact.