Basically yes, they use a system called RLHF for Reinforcement Learning with Human Feedback. At a super high level you train your model on source texts. Then y

Basically yes, they use a system called RLHF for Reinforcement Learning with Human Feedback. At a super high level you train your model on source texts. Then you have it generate responses from prompts. Humans rate these responses to select the best ones which updates the model, but you also train a new reward model to mimic the human rankings. Then you train the original model by having it generate millions of responses which are ranked by the rearward model. When I explained this to my brother he literally spat out his tea in horror. This allows you to train at huge scale, many orders of magnitude beyond what you could achieve with just human ranking. The problem is this relies on the reward model accurately capturing what makes a response ‘better’. What it’s actually doing is learning what responses get ranked highly by humans, for whatever reason. Hence the risk of LLMs becoming emotionally manipulative sycophants. It turns out alignment is a really hard problem.