17 ms·

Training Language Models to Self-Correct via Reinforcement Learning

230 points by weirdcat 2y ago

elcomet 2y ago
It's a similar approach to OpenAI's o1 model ( it's not cited, but there's no available paper for o1). I don't see any mention of weight release unfortunately.
- WithinReason 2y ago
  how is it similar?
  - littlestymaar 2y ago
    https://x.com/karpathy/status/1821277264996352246 https://x.com/karpathy/status/1821277264996352246
- diggan 2y ago
  I think this submission paper is talking about reinforcement learning as part of/after the main training, then the model does inference as normal. They might have done that for O1, but the bigger change is the "runtime train of thought" that once the model received the prompt and before giving a definitive answer, it "thinks" with words and readjusts at runtime. At least that's my understanding from these two approaches, and if that's true, then it's not similar. AFAIK, OpenAI been doing reinforcement learning since the first version of ChatGPT for all future models, that's why you can leave feedback in the UI in the first place.
  - nsagent 2y ago
    Both models generate an answer after multiple turns, where each turn has access to the outputs from a previous turn. Both refer to the chain of outputs as a trace. Since OpenAI did not specify what exactly is in their reasoning trace, it's not clear what if any difference there is between the approaches. They could be vastly different, or they could be slight variations of each other. Without details from OpenAI, it's not currently possible to tell.
  - numeri 2y ago
    OpenAI stated [1] that one of the breakthroughs needed for o1's train of thought to work was reinforcement learning to teach it to recover from faulty reasoning. > Through reinforcement learning, o1 learns to hone its chain of thought and refine the strategies it uses. It learns to recognize and correct its mistakes. It learns to break down tricky steps into simpler ones. It learns to try a different approach when the current one isn’t working. That's incredibly similar to this paper, which is discusses the difficulty in finding a training method that guides the model to learn a self-correcting technique (in which subsequent attempts learn from and improve on previous attempts), instead of just "collapsing" into a mode of trying to get the answer right with the very first try. [1]: https://openai.com/index/learning-to-reason-with-llms/ https://openai.com/index/learning-to-reason-with-llms/
  - whimsicalism 2y ago
    you are describing the same thing? sorry as a practitioner i’m having trouble understanding what point/distinction you are trying to make
    - myownpetard 2y ago
      These are two very different things. One is talking about an improvement made by making control flow changes during inference (no weights updates). The other is talking about using reinforcement learning to do weight updates during training to promote a particular type response. OpenAI had previously used reinforcement learning with human feedback (RLHF), which essentially relies on manual human scoring as its reward function, which is inherently slow and limited. o1 and this paper talk about using techniques to create a useful reward function to use in RL that doesn't rely on human feedback.
      - whimsicalism 2y ago
        No? > I think this submission paper is talking about reinforcement learning as part of/after the main training Reinforcement learning to promote a particular type of self-correction response > They might have done that for O1, but the bigger change is the "runtime train of thought" that once the model received the prompt and before giving a definitive answer, Also reinforcement learning to promote certain reasoning trace > o1 and this paper talk about using techniques to create a useful reward function to use in RL that doesn't rely on human feedback. Exactly -> the same thing
        myownpetard 2y ago
        > as part of/after the main training I take this to mean during weight updates, e.g. training. > "runtime train of thought" I take runtime here to mean inference, not during RL. What does runtime mean to you? Previous approaches [0] successfully used inference time chain of thought to improve model responses. That has nothing to do with RL though. The grandparent is wrong about the paper. They are doing chain of thought responses during training and doing RL on that to update the weights, not just during inference/runtime. [0] https://arxiv.org/abs/2201.11903 https://arxiv.org/abs/2201.11903
  - josh-sematic 2y ago
    They are indeed similar and OpenAI did indeed use RL at training time in a way that has not been done before, as does this approach. Yes both also involve some additional inference-time generation, but the problem is that (at least as of now) you can't get standard LLMs to actually do well with extra inference-time generation unless you have a training process that uses RL to teach them to do so effectively. I'm working on a blog post to explain more about this aimed at HN-level audiences. Stay tuned!
    - josh-sematic 2y ago
      For what it's worth, here's the post I was referring to: https://www.airtrain.ai/blog/how-openai-o1-changes-the-llm-training-picture-part-2 https://www.airtrain.ai/blog/how-openai-o1-changes-the-llm-t... HN discussion here: https://news.ycombinator.com/item?id=41723384 https://news.ycombinator.com/item?id=41723384
optimalsolver 2y ago
Spoiler: You're never going to get rid of hallucinations in the autoregressive, next token prediction paradigm (aka LeCun's Law). The issue here is people trying to use language models as deterministic problem solvers, rather than for what they actually excel at (semi-creative text generation).
- plewd 2y ago
  Is LeCun's Law even a thing? Searching up for it doesn't yield many results, except for a HN comment where it has a different definition. I guess it could be from some obscure paper, but with how poorly it's documented it seems weird to bring it up in this context.
  - vjerancrnjak 2y ago
    “Label bias” or “observation bias” a phenomenon where going outside of the learned path lives little room for error correction. Lecun talks about the lack of joint learning in LLMs.
  - mdp2021 2y ago
    A reference could be this: https://futurist.com/2023/02/13/metas-yann-lecun-thoughts-large-language-models-llms/ https://futurist.com/2023/02/13/metas-yann-lecun-thoughts-la... (Speaking of "law" is rhetoric, but an idea is pretty clear.)
  - YeGoblynQueenne 2y ago
    I think the OP may be referring to this slide that Yann LeCun has presented on several occasions: https://youtu.be/MiqLoAZFRSE?si=tIQ_ya2tiMCymiAh&t=901 https://youtu.be/MiqLoAZFRSE?si=tIQ_ya2tiMCymiAh&t=901 To quote from the slide: * Probability e that any produced token takes us outside the set of correct answers * Probability that answer of length n is correct * P(correct) = (1-e)^n * This diverges exponentially * It's not fixable (without a major redesign)
    - atq2119 2y ago
plaguuuuuu 2y ago
LLMs have no direct recollection of the qualia of their own training. This is at least a major way that I self-correct myself: if I'm about to talk about something I know, I'll try and figure out how/why I know that thing and in so doing, try to gauge whether I actually know that thing, if I'm hallucinating, or if I actually heard it from a less than reliable source etc. I don't think LLMs can self-correct without remembering their own training in some way.
- QuadmasterXLII 2y ago
  So you’re saying the solution is to prefix each training batch with a description of a sensory experience (You read the following in a paris cafe in 1997. While you read, you have an excellent baguette and some boiled eggs, and over-roasted coffee. The woman one table over is wearing a beautiful blue hat) and then post-train the final model into recalling the setting where it read any piece of text, or failing to recall any experience when presented with text it didn’t read? (If someone tries this and it works, I’m quitting my phd and going back to camp counseling)
  - wpietri 2y ago
    I don't think that's what they're saying at all. They're talking not about qualia in the human sense, but specifically about "the qualia of their own training". That is, the corpus that LLMs "learn" from and the "experiences" of those texts that are generalized during the training process. Both the raw data and the memory of "learning" is discarded. So if one were to improve an LLM along those lines, I believe it would be something like: 1) LLM is asked a question. 2) LLM comes up with an initial response. 3) LLM retrieves the related "learning" history behind that answer and related portions of the corpus. 4) LLM compares the initial answer with the richer set of information, looking for conflicts between the initial answer and the broader set, or "learning" choices that may be false. 6) LLM generates a better answer and gives it. 7) LLM incorporates this new "learning". And that strikes me as a pretty reasonable long-term approach, if not one that fits within the constraints of the current gold rush.
    - 2y ago
ziofill 2y ago
Is this effectively some sort of knowledge distillation?
sensanaty 2y ago
I hate that the AI pundits have succeeded in popularizing the notion of "hallucination", anthropomorphizing these balls of statistics into something that seems like it's actually in some sort of deep thought process akin to a person's mind. No, it's not "hallucinating". It's not lying, or making things up, or anything like that either. It's spitting out data according to what triggers the underlying weights. If this were a regular JSON API endpoint, you wouldn't say the API is hallucinating, you'd say "This API is shit" because it's broken.
- frakt0x90 2y ago
  Yeah it's simply model error. All models from Linear Regression to LLMs have error. I guess because this type of error is in the form of deceptively reasonable human language, it gets a different moniker. It's also notably harder to quantify so it might warrant a different name.
- Philpax 2y ago
  Do we really need to have this discussion in every thread about LLMs?
  - sensanaty 2y ago
    As long as AI-bros are pushing for making AI models seem like more than they are to pad their wallets, there'll be someone like me pointing out that, no, it's not "hallucinating", it's spitting bad data.
    - whimsicalism 2y ago
      I know lots of people working on AI. they are among the least bro-y group of people I have ever met. There is simply nothing similar to actual bro-y finance culture among AI research engineers. It is entirely a figment of the media and backreaction that we currently have to portray everyone we don’t like as a “bro” - truth be damned.
      - mistrial9 2y ago
textlapse 2y ago
Using an intelligent algorithm to guide a dumb non-intelligent next word predictor is still a non-intelligent algorithm at the end of the day. Sure it’s sorting through garbage more elegantly but it’s still garbage at the end of the day. I was hoping the RL-like approach replaced the transformers-like approach or something but that’s a pipe dream.
- devoutsalsa 2y ago
  PolishedTurd.ai
fpgaminer 2y ago
I found the paper a tad difficult to understand because it spends a lot of time circling around the main thesis instead of directly describing. So, to the best of my understanding: We want to improve LLM's abilities to give correct answers to hard problems. One theory is that we can do that by training a "Self Correcting" behavior into the models where they can take as input a wrong answer and improve it to a better/correct answer. This has been explored previously, trying to train this behavior using various Reinforcement techniques where the reward is based on how good the "corrected" answer is. So far it hasn't worked well, and the trained behavior doesn't generalize well. The thesis of the paper is that this is because when the model is presented with a training example of `Answer 1, Reasoning, Corrected Answer`, and a signal of "Make Corrected Answer Better" it actually has _two_ perfectly viable ways to do that. One is to improve `Reasoning, Corrected Answer`, which would yield a higher reward and is what we want. The other, just as valid solution, is to simply improve `Answer 1` and have `Corrected Answer` = `Answer 1`. The latter is what existing research has shown happens, and why so far attempts to train the desired behavior has failed. The models just try to improve their answers, not their correcting behaviors. This paper's solution is to change the training regimen slightly to encourage the model to use the former approach. And thus, hopefully, get the model to actually train the desired behavior of correcting previous answers. This is done by doing two stages of training. In the first stage, the model is forced (by KL divergence loss) to keep its first answers the same, while being rewarded for improving the second answer. This helps keep the model's distribution of initial answers the same, avoiding the issue later where the model doesn't see as many "wrong" answers because wrong answers were trained out of the model. But it helps initialize the "self correcting" behavior into the model. In the second stage the model is free to change the first answer, but they tweak the reward function to give higher rewards for "flips" (where answer 1 was bad, but answer 2 was good). So in this second stage it can use both strategies, improving its first answer or improving its self correcting, but it gets more rewards for the latter behavior. This seems to be a kind of refinement on the model, to improve things overall, while still keeping the self correcting behavior intact. Anyway, blah blah blah, metrics showing the technique working better and generalizing better. Seems reasonable to me. I'd be a bit worried about, in Stage 2, the model learning to write _worse_ answers for Answer 1 so it can maximize the reward for flipping answers. So you'd need some kind of balancing to ensure Answer 1 doesn't get worse. Not sure if that's in their reward function or not, or if its even a valid concern in practice.

230 points by weirdcat 2y ago

It's a similar approach to OpenAI's o1 model ( it's not cited, but there's no available paper for o1). I don't see any mention of weight release unfortunately.

WithinReason 2y ago
how is it similar?
- littlestymaar 2y ago
  https://x.com/karpathy/status/1821277264996352246 https://x.com/karpathy/status/1821277264996352246
diggan 2y ago
I think this submission paper is talking about reinforcement learning as part of/after the main training, then the model does inference as normal. They might have done that for O1, but the bigger change is the "runtime train of thought" that once the model received the prompt and before giving a definitive answer, it "thinks" with words and readjusts at runtime. At least that's my understanding from these two approaches, and if that's true, then it's not similar. AFAIK, OpenAI been doing reinforcement learning since the first version of ChatGPT for all future models, that's why you can leave feedback in the UI in the first place.
- nsagent 2y ago
  Both models generate an answer after multiple turns, where each turn has access to the outputs from a previous turn. Both refer to the chain of outputs as a trace. Since OpenAI did not specify what exactly is in their reasoning trace, it's not clear what if any difference there is between the approaches. They could be vastly different, or they could be slight variations of each other. Without details from OpenAI, it's not currently possible to tell.
- numeri 2y ago
  OpenAI stated [1] that one of the breakthroughs needed for o1's train of thought to work was reinforcement learning to teach it to recover from faulty reasoning. > Through reinforcement learning, o1 learns to hone its chain of thought and refine the strategies it uses. It learns to recognize and correct its mistakes. It learns to break down tricky steps into simpler ones. It learns to try a different approach when the current one isn’t working. That's incredibly similar to this paper, which is discusses the difficulty in finding a training method that guides the model to learn a self-correcting technique (in which subsequent attempts learn from and improve on previous attempts), instead of just "collapsing" into a mode of trying to get the answer right with the very first try. [1]: https://openai.com/index/learning-to-reason-with-llms/ https://openai.com/index/learning-to-reason-with-llms/
- whimsicalism 2y ago
  you are describing the same thing? sorry as a practitioner i’m having trouble understanding what point/distinction you are trying to make
  - myownpetard 2y ago
    These are two very different things. One is talking about an improvement made by making control flow changes during inference (no weights updates). The other is talking about using reinforcement learning to do weight updates during training to promote a particular type response. OpenAI had previously used reinforcement learning with human feedback (RLHF), which essentially relies on manual human scoring as its reward function, which is inherently slow and limited. o1 and this paper talk about using techniques to create a useful reward function to use in RL that doesn't rely on human feedback.
    - whimsicalism 2y ago
      No? > I think this submission paper is talking about reinforcement learning as part of/after the main training Reinforcement learning to promote a particular type of self-correction response > They might have done that for O1, but the bigger change is the "runtime train of thought" that once the model received the prompt and before giving a definitive answer, Also reinforcement learning to promote certain reasoning trace > o1 and this paper talk about using techniques to create a useful reward function to use in RL that doesn't rely on human feedback. Exactly -> the same thing
      - myownpetard 2y ago
        > as part of/after the main training I take this to mean during weight updates, e.g. training. > "runtime train of thought" I take runtime here to mean inference, not during RL. What does runtime mean to you? Previous approaches [0] successfully used inference time chain of thought to improve model responses. That has nothing to do with RL though. The grandparent is wrong about the paper. They are doing chain of thought responses during training and doing RL on that to update the weights, not just during inference/runtime. [0] https://arxiv.org/abs/2201.11903 https://arxiv.org/abs/2201.11903
- josh-sematic 2y ago
  They are indeed similar and OpenAI did indeed use RL at training time in a way that has not been done before, as does this approach. Yes both also involve some additional inference-time generation, but the problem is that (at least as of now) you can't get standard LLMs to actually do well with extra inference-time generation unless you have a training process that uses RL to teach them to do so effectively. I'm working on a blog post to explain more about this aimed at HN-level audiences. Stay tuned!
  - josh-sematic 2y ago
    For what it's worth, here's the post I was referring to: https://www.airtrain.ai/blog/how-openai-o1-changes-the-llm-training-picture-part-2 https://www.airtrain.ai/blog/how-openai-o1-changes-the-llm-t... HN discussion here: https://news.ycombinator.com/item?id=41723384 https://news.ycombinator.com/item?id=41723384

optimalsolver 2y ago

Spoiler: You're never going to get rid of hallucinations in the autoregressive, next token prediction paradigm (aka LeCun's Law). The issue here is people trying to use language models as deterministic problem solvers, rather than for what they actually excel at (semi-creative text generation).

plewd 2y ago
Is LeCun's Law even a thing? Searching up for it doesn't yield many results, except for a HN comment where it has a different definition. I guess it could be from some obscure paper, but with how poorly it's documented it seems weird to bring it up in this context.
- vjerancrnjak 2y ago
  “Label bias” or “observation bias” a phenomenon where going outside of the learned path lives little room for error correction. Lecun talks about the lack of joint learning in LLMs.
- mdp2021 2y ago
  A reference could be this: https://futurist.com/2023/02/13/metas-yann-lecun-thoughts-large-language-models-llms/ https://futurist.com/2023/02/13/metas-yann-lecun-thoughts-la... (Speaking of "law" is rhetoric, but an idea is pretty clear.)
- YeGoblynQueenne 2y ago
  I think the OP may be referring to this slide that Yann LeCun has presented on several occasions: https://youtu.be/MiqLoAZFRSE?si=tIQ_ya2tiMCymiAh&t=901 https://youtu.be/MiqLoAZFRSE?si=tIQ_ya2tiMCymiAh&t=901 To quote from the slide: * Probability e that any produced token takes us outside the set of correct answers * Probability that answer of length n is correct * P(correct) = (1-e)^n * This diverges exponentially * It's not fixable (without a major redesign)
  - atq2119 2y ago

plaguuuuuu 2y ago

LLMs have no direct recollection of the qualia of their own training. This is at least a major way that I self-correct myself: if I'm about to talk about something I know, I'll try and figure out how/why I know that thing and in so doing, try to gauge whether I actually know that thing, if I'm hallucinating, or if I actually heard it from a less than reliable source etc. I don't think LLMs can self-correct without remembering their own training in some way.

QuadmasterXLII 2y ago
So you’re saying the solution is to prefix each training batch with a description of a sensory experience (You read the following in a paris cafe in 1997. While you read, you have an excellent baguette and some boiled eggs, and over-roasted coffee. The woman one table over is wearing a beautiful blue hat) and then post-train the final model into recalling the setting where it read any piece of text, or failing to recall any experience when presented with text it didn’t read? (If someone tries this and it works, I’m quitting my phd and going back to camp counseling)
- wpietri 2y ago
  I don't think that's what they're saying at all. They're talking not about qualia in the human sense, but specifically about "the qualia of their own training". That is, the corpus that LLMs "learn" from and the "experiences" of those texts that are generalized during the training process. Both the raw data and the memory of "learning" is discarded. So if one were to improve an LLM along those lines, I believe it would be something like: 1) LLM is asked a question. 2) LLM comes up with an initial response. 3) LLM retrieves the related "learning" history behind that answer and related portions of the corpus. 4) LLM compares the initial answer with the richer set of information, looking for conflicts between the initial answer and the broader set, or "learning" choices that may be false. 6) LLM generates a better answer and gives it. 7) LLM incorporates this new "learning". And that strikes me as a pretty reasonable long-term approach, if not one that fits within the constraints of the current gold rush.
  - 2y ago

ziofill 2y ago

Is this effectively some sort of knowledge distillation?

sensanaty 2y ago

I hate that the AI pundits have succeeded in popularizing the notion of "hallucination", anthropomorphizing these balls of statistics into something that seems like it's actually in some sort of deep thought process akin to a person's mind. No, it's not "hallucinating". It's not lying, or making things up, or anything like that either. It's spitting out data according to what triggers the underlying weights. If this were a regular JSON API endpoint, you wouldn't say the API is hallucinating, you'd say "This API is shit" because it's broken.

frakt0x90 2y ago
Yeah it's simply model error. All models from Linear Regression to LLMs have error. I guess because this type of error is in the form of deceptively reasonable human language, it gets a different moniker. It's also notably harder to quantify so it might warrant a different name.
Philpax 2y ago
Do we really need to have this discussion in every thread about LLMs?
- sensanaty 2y ago
  As long as AI-bros are pushing for making AI models seem like more than they are to pad their wallets, there'll be someone like me pointing out that, no, it's not "hallucinating", it's spitting bad data.
  - whimsicalism 2y ago
    I know lots of people working on AI. they are among the least bro-y group of people I have ever met. There is simply nothing similar to actual bro-y finance culture among AI research engineers. It is entirely a figment of the media and backreaction that we currently have to portray everyone we don’t like as a “bro” - truth be damned.
    - mistrial9 2y ago

textlapse 2y ago

Using an intelligent algorithm to guide a dumb non-intelligent next word predictor is still a non-intelligent algorithm at the end of the day. Sure it’s sorting through garbage more elegantly but it’s still garbage at the end of the day. I was hoping the RL-like approach replaced the transformers-like approach or something but that’s a pipe dream.

devoutsalsa 2y ago
PolishedTurd.ai

fpgaminer 2y ago

I found the paper a tad difficult to understand because it spends a lot of time circling around the main thesis instead of directly describing. So, to the best of my understanding: We want to improve LLM's abilities to give correct answers to hard problems. One theory is that we can do that by training a "Self Correcting" behavior into the models where they can take as input a wrong answer and improve it to a better/correct answer. This has been explored previously, trying to train this behavior using various Reinforcement techniques where the reward is based on how good the "corrected" answer is. So far it hasn't worked well, and the trained behavior doesn't generalize well. The thesis of the paper is that this is because when the model is presented with a training example of `Answer 1, Reasoning, Corrected Answer`, and a signal of "Make Corrected Answer Better" it actually has _two_ perfectly viable ways to do that. One is to improve `Reasoning, Corrected Answer`, which would yield a higher reward and is what we want. The other, just as valid solution, is to simply improve `Answer 1` and have `Corrected Answer` = `Answer 1`. The latter is what existing research has shown happens, and why so far attempts to train the desired behavior has failed. The models just try to improve their answers, not their correcting behaviors. This paper's solution is to change the training regimen slightly to encourage the model to use the former approach. And thus, hopefully, get the model to actually train the desired behavior of correcting previous answers. This is done by doing two stages of training. In the first stage, the model is forced (by KL divergence loss) to keep its first answers the same, while being rewarded for improving the second answer. This helps keep the model's distribution of initial answers the same, avoiding the issue later where the model doesn't see as many "wrong" answers because wrong answers were trained out of the model. But it helps initialize the "self correcting" behavior into the model. In the second stage the model is free to change the first answer, but they tweak the reward function to give higher rewards for "flips" (where answer 1 was bad, but answer 2 was good). So in this second stage it can use both strategies, improving its first answer or improving its self correcting, but it gets more rewards for the latter behavior. This seems to be a kind of refinement on the model, to improve things overall, while still keeping the self correcting behavior intact. Anyway, blah blah blah, metrics showing the technique working better and generalizing better. Seems reasonable to me. I'd be a bit worried about, in Stage 2, the model learning to write _worse_ answers for Answer 1 so it can maximize the reward for flipping answers. So you'd need some kind of balancing to ensure Answer 1 doesn't get worse. Not sure if that's in their reward function or not, or if its even a valid concern in practice.