4 ms·

> All user facing LLMs go through Reinforcement Learning. Contrary to popular belief, RL's _primary_ purpose isn't to "align" them to make them "safe." It's to

by gwern 2y ago

> All user facing LLMs go through Reinforcement Learning. Contrary to popular belief, RL's _primary_ purpose isn't to "align" them to make them "safe." It's to make them actually usable. Are you claiming that non-myopic token prediction emerges solely from RL, and if Anthropic does this analysis on Claude before RL training (or if one examines other models where no RLHF was done, such as old GPT-2 checkpoints), none of these advance prediction mechanisms will exist?

fpgaminer 2y ago
No, it probably exists in the raw LLM and gets both significantly strengthened and has its range extended. Such that it dominates the model's behavior, making it several orders of magnitude more reliable in common usage. Kinda of like how "reasoning" exists in a weak, short range way in non-reasoning models. With RL that encourages reasoning, that machinery gets brought to the forefront and becomes more complex and capable.
- rafaelero 2y ago
  So why did you feel the need to post that next-token prediction is not the reason this behavior emerge?
rcxdude 2y ago
Another important aspect of the RL process is that it's fine-tuning with some feedback on the quality of data: a 'raw' LLM has been trained on a lot of very low-quality data, and it has an incentive to predict that accurately as well, because there's no means to effectively rate a copy of most of the text on the internet. So there's a lot of biases in the model which basically mean it will include low-quality predictions in a given 'next token' estimate, because if it doesn't it will get penalised when it is fed the low quality data during the training. With RLHF it gets a signal during training for whether the next token it's trying to predict are part of a 'good' response or a 'bad' response, so it can learn to suppress features it learned in the first part of the process which are not useful. (you seem the same with image generators: they've been trained on a bunch of very nice-looking art and photos, but they've also been trained on triply-compressed badly cropped memes and terrible MS-paint art. You need to have a plan for getting the model to output the former and not the latter if you want it to be useful)

by gwern 2y ago

fpgaminer 2y ago

No, it probably exists in the raw LLM and gets both significantly strengthened and has its range extended. Such that it dominates the model's behavior, making it several orders of magnitude more reliable in common usage. Kinda of like how "reasoning" exists in a weak, short range way in non-reasoning models. With RL that encourages reasoning, that machinery gets brought to the forefront and becomes more complex and capable.

rafaelero 2y ago
So why did you feel the need to post that next-token prediction is not the reason this behavior emerge?

rcxdude 2y ago

Another important aspect of the RL process is that it's fine-tuning with some feedback on the quality of data: a 'raw' LLM has been trained on a lot of very low-quality data, and it has an incentive to predict that accurately as well, because there's no means to effectively rate a copy of most of the text on the internet. So there's a lot of biases in the model which basically mean it will include low-quality predictions in a given 'next token' estimate, because if it doesn't it will get penalised when it is fed the low quality data during the training. With RLHF it gets a signal during training for whether the next token it's trying to predict are part of a 'good' response or a 'bad' response, so it can learn to suppress features it learned in the first part of the process which are not useful. (you seem the same with image generators: they've been trained on a bunch of very nice-looking art and photos, but they've also been trained on triply-compressed badly cropped memes and terrible MS-paint art. You need to have a plan for getting the model to output the former and not the latter if you want it to be useful)