3 ms·
Most of the other replies to you, except for the one by tempusalaria, are not really answering the question. Broadly, while there was a lot of initial exciteme
by whimsicalism 2y ago
Most of the other replies to you, except for the one by tempusalaria, are not really answering the question.
Broadly, while there was a lot of initial excitement - it simply does not seem like offline + off-policy RL can beat online + on-policy RL methods like PPO. Sampling trajectories from the actual model you are training and scoring them seems like it works much better in practice, never mind the additional flexibility methods like PPO provide over the form of the reward function.
- eggie5 2y agoWhat's _online_ RL for an LLM? Saw this on the llama 3.3 reports too...
- whimsicalism 2y agoOnline RL for LLMs means you are sampling from the model, scoring immediately, and passing gradients back to the model. As opposed to, sampling from the model a bunch, getting scores offline, and then fine tuning the model on those offline scored generations.