2 ms·
It's been fine-tuned to put on a "helpful assistant face." Given the corpora, it probably has been trained explicitly on the pronoun question [I doubt it is tha
by whimsicalism 4y ago
It's been fine-tuned to put on a "helpful assistant face." Given the corpora, it probably has been trained explicitly on the pronoun question [I doubt it is that uncommon], but will also just put on this face for any generic question.
- int_19h 4y agoThis stuff is not fine-tuning - it's RLHF (reinforcement learning from human feedback). Basically just a lot of people marking up responses as good / bad according to the assessment criteria in the script they're given. And yes, it is very likely that it would have seen that exactly question in RLHF. But even if not, it had seen enough to broadly "understand" what kinds of topics are sensitive and how to tiptoe around them.
- whimsicalism 4y agoRLHF is fine tuning :)
- int_19h 4y agoYou're right, my apologies. What I meant is that specifically in ChatGPT context they have been always talking about RHLF as separate from fine-tuning on preassembled training data. As I understand, they use the latter mainly to get the desired behavior as a chatbot "eager" to answer questions and solve tasks. And then RLHF is what "puts the smiley face on top", so that it refuses to answer some questions, or gives "aligned" answers. The pronoun would be in that category.
- whimsicalism 4y agoMy bet is that they use the pre-assembled training data to train the reward LLM and maybe also for fine tuning the model in addition to RLHF. This is my best guess, afaik the research is pretty opaque right now.