4 ms·
LLM just predicts the next token given the previous tokens(this can be trained without manual labelling by humans). Instruct GPT and ChatGPT use reinforcement
by isaacfung 4y ago
LLM just predicts the next token given the previous tokens(this can be trained without manual labelling by humans).
Instruct GPT and ChatGPT use reinforcement learning from human feedback to align the model with human intents so it understands instructions.
https://huggingface.co/blog/rlhf https://huggingface.co/blog/rlhf
- bilsbie 4y agoThanks. So what does the output look like without rlhf?
- simonw 4y agoThis comment has a useful comparison between the two: https://news.ycombinator.com/item?id=35140447 https://news.ycombinator.com/item?id=35140447
- valine 4y agoIt can look like anything. Sometimes it will answer your questions, other times it will continue the question like its the one asking. I've also seen it randomly output footers and copyright like it just got to the end of a webpage. Its makes sense when you think about how the training data is random text on the internet. Sometimes the most likely next token is the end of a webpage after an unanswered question.
- sanxiyn 4y agoNote that Alpaca is NOT using RLHF. It explicitly states it used supervised finetuning.
- est 4y agoIt says > We train the Alpaca model on 52K instruction-following demonstrations generated in the style of self-instruct using text-davinci-003 Which leads to self-instruct https://github.com/yizhongw/self-instruct https://github.com/yizhongw/self-instruct From a glimpse they used a LM to classify instructions & train the model which IMHO is very similar to RLHF
- sanxiyn 4y agoNo, it is not RLHF because there is no reward model involved. See also OpenAI's explanation here: https://platform.openai.com/docs/model-index-for-researchers https://platform.openai.com/docs/model-index-for-researchers