2 ms·
I just added a new section to my post about some clues from the InstructGPT paper: https://simonwillison.net/2023/Jun/4/closed-model-training/#instructgpt https
by simonw 3y ago
I just added a new section to my post about some clues from the InstructGPT paper: https://simonwillison.net/2023/Jun/4/closed-model-training/#instructgpt https://simonwillison.net/2023/Jun/4/closed-model-training/#...
That paper (in January 2022) says:
> To make our models safer, more helpful, and more aligned, we use an existing technique called reinforcement learning from human feedback (RLHF). On prompts submitted by our customers to the API[A] our labelers provide demonstrations of the desired model behavior, and rank several outputs from our models. We then use this data to fine-tune GPT-3.
This was before they introduced their policy that data submitted to the API would not be used in any way to affect future training (that was in March this year).
Personally I think there's a significant difference between "your input to the model will be used as raw input for future pre-training" and "your prompts to the model will be used in an exercise where human labelers are shown multiple responses and asked to pick the best one".
What I'd really like here is for OpenAI to help people understand how that data is being used in as much detail as possible.
I guess the problem with that is it makes it harder for them to use that data in new ways they haven't yet anticipated needing in the future.