4 ms·
Why were human data labelers involved in the creation of ChatGPT? I thought the approach was to let the neural network predict the next word in a large corpus
by TekMol 3y ago
Why were human data labelers involved in the creation of ChatGPT?
I thought the approach was to let the neural network predict the next word in a large corpus of text?
- oefrha 3y agoIf you don’t classify content that should be filtered out you quickly end up with a Microsoft Tay-like fiasco.
- spott 3y agoRlhf has a human labeling component: given two outputs to the same prompt, which one is better?
- mlinhares 3y agoYou need people to evaluate the answers to improve the output and adjust the weights, there is a lot of manual labor involved at training every single one of these models, it’s not just magic.
- famouswaffles 3y agoThere doesn't have to be manual labor. Anthropic (the Claude Models) use reinforcement learning from AI feedback (dubbed Constitutional AI) and it works great too.
- version_five 3y agoI don't know the scope of OpenAIs data cleaning or RLHF efforts. But I'd guess this is a big part of what's made them the leader. It's one thing to train an LLM on a big text corpus, as researchers do. It's another to actually wade into it and do the curation to turn it into a real product. I know there are public instruction datasets out there, but my guess is OpenAI has put a lot of money into getting cleaner data and more supervised / feedback data. Doing that extra work is the kind if thing that wins when building a product.
- famouswaffles 3y agoIt's part of it but it's not necessary to get humans to do it. If you're aware of the Claude Models, they use reinforcement learning from AI feedback and they're great. The best Claude model is only behind GPT-4
- famouswaffles 3y agoAfter a large model is pretrained on a large corpus, it's not by default an instruction follower. You need an additional fine-tuning process to get it to follow instructions. There are a number of ways to do that. 1. Supervised Fine-tuning. That is Fine-tuning on a corpus of instructions and responses 2. Reinforcement learning from Human feedback where the model trains to adjusts its responses based on feedback from humans 3. Reinforcement learning from AI feedback. Like above but with another LLM doing the rating/classifying.