4 ms·

By now, everyone has heard the explanation that ChatGPT is a transformer encoder-decoder that responds to prompts by iteratively predicting the first word in th

by wrp 4y ago

By now, everyone has heard the explanation that ChatGPT is a transformer encoder-decoder that responds to prompts by iteratively predicting the first word in the response, then the second word, and so on... What we need now is explanation of all the further stuff added to that basic capability.

sendfoods 4y ago
Didn't Alpaca attempt to explain and test the "secret sauce"? The RL fine tuning?
visarga 4y ago
The pre-trained model is stage 1 - it has seen everything, but it is wild. If you ask it "What is the capital of US?" it will reply "What is the capital of Canada?"... Stage 2 is task solving practice. We use 1000-2000 supervised datasets, formatted as prompt-input-output texts. They could be anything: translation, sentiment classification, question answering, etc. We also include prompt-code pairs. This teaches the model to solve tasks (it "hires" this ability from the model). Apparently training on code is essential, without it the model doesn't develop reasoning abilities. But still the model is not well behaved, it doesn't answer in a way we like. So in stage 3 it goes to human preference tuning (RLHF). This is based on human preferences between pairs of LLM answers. After RLHF it learns to behave and to abstain from certain topics. You need stage 1 for general knowledge, stage 2 for learning to execute prompts, stage 3 to make it behave.
- wrp 4y ago
  Regarding Stage 2. Are you saying that ChatGPT's facility to recognize and process commands is derived entirely from training on supervised datasets and not hand-crafted logic? Can you point me to any reading on this?
moritzdubois 4y ago
> By now, everyone has heard the explanation that ChatGPT is a transformer encoder-decoder that ... Except it is wrong. GPT models are decoder-only transformers. See Andrej Karpathy's outstanding series on implementing a toy-scale GPT model.

by wrp 4y ago

sendfoods 4y ago

Didn't Alpaca attempt to explain and test the "secret sauce"? The RL fine tuning?

visarga 4y ago

The pre-trained model is stage 1 - it has seen everything, but it is wild. If you ask it "What is the capital of US?" it will reply "What is the capital of Canada?"... Stage 2 is task solving practice. We use 1000-2000 supervised datasets, formatted as prompt-input-output texts. They could be anything: translation, sentiment classification, question answering, etc. We also include prompt-code pairs. This teaches the model to solve tasks (it "hires" this ability from the model). Apparently training on code is essential, without it the model doesn't develop reasoning abilities. But still the model is not well behaved, it doesn't answer in a way we like. So in stage 3 it goes to human preference tuning (RLHF). This is based on human preferences between pairs of LLM answers. After RLHF it learns to behave and to abstain from certain topics. You need stage 1 for general knowledge, stage 2 for learning to execute prompts, stage 3 to make it behave.

wrp 4y ago
Regarding Stage 2. Are you saying that ChatGPT's facility to recognize and process commands is derived entirely from training on supervised datasets and not hand-crafted logic? Can you point me to any reading on this?

moritzdubois 4y ago

> By now, everyone has heard the explanation that ChatGPT is a transformer encoder-decoder that ... Except it is wrong. GPT models are decoder-only transformers. See Andrej Karpathy's outstanding series on implementing a toy-scale GPT model.