4 ms·

Agree, I was blown away by the quality of the generated text until realizing that it was simply too good to be true. The fact that they picked and chose makes f

• gwern 7y ago
  But on the other hand, 'picking and choosing' can be automated in many ways (eg A/B testing), and in a very real sense, that's what the last research OA released on GPT-2 is about: https://openai.com/blog/fine-tuning-gpt-2/ https://openai.com/blog/fine-tuning-gpt-2/ (Because the human preferences are used to train a critic which then 'picks and chooses' by ranking output by quality to further train the text generator.)
  • gwern 7y ago
    I have been experimenting with the preference-learning code for writing poetry, and it does seem to work: I've already noticed improvements in the generated poetry in increasing the amounts of rhymes, avoiding undesirable text genres like Greek/Latin/Spanish/English-prose, and decreasing the amount of repetition. (One of the most irritating and longstanding failure modes in neural text generation is that it often degenerates into repeating the same phrase many times, so it's pleasing to see that penalizing this in ratings does seem to then cause the generator to learn to avoid it.) Unfortunately, model-free RL training using PPO is extremely slow. Fortunately, I can't see any reason that it has to be PPO or model-free RL at all, since the reward model ought to be fully differentiable and allow calculation of true gradients / model-based RL, so OOM speedups ought to be possible quite easily.